2023-04-12 13:53:31,124 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00 2023-04-12 13:53:31,140 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-12 13:53:31,158 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-12 13:53:31,158 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/cluster_34daed27-f3ec-9fef-aef6-6c33478889d6, deleteOnExit=true 2023-04-12 13:53:31,158 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-12 13:53:31,159 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/test.cache.data in system properties and HBase conf 2023-04-12 13:53:31,159 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/hadoop.tmp.dir in system properties and HBase conf 2023-04-12 13:53:31,160 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/hadoop.log.dir in system properties and HBase conf 2023-04-12 13:53:31,160 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-12 13:53:31,161 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-12 13:53:31,161 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-12 13:53:31,286 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-12 13:53:31,703 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-12 13:53:31,708 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-12 13:53:31,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-12 13:53:31,709 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-12 13:53:31,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-12 13:53:31,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-12 13:53:31,710 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-12 13:53:31,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-12 13:53:31,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-12 13:53:31,711 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-12 13:53:31,712 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/nfs.dump.dir in system properties and HBase conf 2023-04-12 13:53:31,712 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/java.io.tmpdir in system properties and HBase conf 2023-04-12 13:53:31,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-12 13:53:31,713 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-12 13:53:31,714 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-12 13:53:32,304 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-12 13:53:32,309 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-12 13:53:32,618 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-12 13:53:32,804 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-12 13:53:32,826 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 13:53:32,870 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 13:53:32,903 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/java.io.tmpdir/Jetty_localhost_43887_hdfs____45jq4i/webapp 2023-04-12 13:53:33,061 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43887 2023-04-12 13:53:33,103 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-12 13:53:33,103 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-12 13:53:33,540 WARN [Listener at localhost/34129] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 13:53:33,708 WARN [Listener at localhost/34129] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-12 13:53:33,728 WARN [Listener at localhost/34129] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 13:53:33,736 INFO [Listener at localhost/34129] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 13:53:33,742 INFO [Listener at localhost/34129] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/java.io.tmpdir/Jetty_localhost_44285_datanode____940u52/webapp 2023-04-12 13:53:33,863 INFO [Listener at localhost/34129] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44285 2023-04-12 13:53:34,259 WARN [Listener at localhost/46495] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 13:53:34,829 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4d7267c69d84ac6c: Processing first storage report for DS-33248ea2-084d-465e-ba85-b2841516c3d6 from datanode 06037e37-fbd1-4b2b-b8f8-c56f07685505 2023-04-12 13:53:34,832 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4d7267c69d84ac6c: from storage DS-33248ea2-084d-465e-ba85-b2841516c3d6 node DatanodeRegistration(127.0.0.1:38341, datanodeUuid=06037e37-fbd1-4b2b-b8f8-c56f07685505, infoPort=41465, infoSecurePort=0, ipcPort=46495, storageInfo=lv=-57;cid=testClusterID;nsid=884192034;c=1681307612384), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-12 13:53:34,832 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4d7267c69d84ac6c: Processing first storage report for DS-1661dde4-1909-4536-b072-eeaf93ff1e43 from datanode 06037e37-fbd1-4b2b-b8f8-c56f07685505 2023-04-12 13:53:34,832 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4d7267c69d84ac6c: from storage DS-1661dde4-1909-4536-b072-eeaf93ff1e43 node DatanodeRegistration(127.0.0.1:38341, datanodeUuid=06037e37-fbd1-4b2b-b8f8-c56f07685505, infoPort=41465, infoSecurePort=0, ipcPort=46495, storageInfo=lv=-57;cid=testClusterID;nsid=884192034;c=1681307612384), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-12 13:53:34,856 DEBUG [Listener at localhost/46495] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00 2023-04-12 13:53:34,926 INFO [Listener at localhost/46495] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/cluster_34daed27-f3ec-9fef-aef6-6c33478889d6/zookeeper_0, clientPort=58527, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/cluster_34daed27-f3ec-9fef-aef6-6c33478889d6/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/cluster_34daed27-f3ec-9fef-aef6-6c33478889d6/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-12 13:53:34,942 INFO [Listener at localhost/46495] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58527 2023-04-12 13:53:34,953 INFO [Listener at localhost/46495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:34,955 INFO [Listener at localhost/46495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:35,631 INFO [Listener at localhost/46495] util.FSUtils(471): Created version file at hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964 with version=8 2023-04-12 13:53:35,631 INFO [Listener at localhost/46495] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/hbase-staging 2023-04-12 13:53:35,933 INFO [Listener at localhost/46495] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-12 13:53:36,402 INFO [Listener at localhost/46495] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 13:53:36,434 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 13:53:36,435 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 13:53:36,435 INFO [Listener at localhost/46495] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 13:53:36,435 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 13:53:36,435 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 13:53:36,580 INFO [Listener at localhost/46495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 13:53:36,666 DEBUG [Listener at localhost/46495] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-12 13:53:36,781 INFO [Listener at localhost/46495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39787 2023-04-12 13:53:36,792 INFO [Listener at localhost/46495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:36,795 INFO [Listener at localhost/46495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:36,817 INFO [Listener at localhost/46495] zookeeper.RecoverableZooKeeper(93): Process identifier=master:39787 connecting to ZooKeeper ensemble=127.0.0.1:58527 2023-04-12 13:53:36,858 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:397870x0, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 13:53:36,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:39787-0x10178a628840000 connected 2023-04-12 13:53:36,889 DEBUG [Listener at localhost/46495] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 13:53:36,890 DEBUG [Listener at localhost/46495] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 13:53:36,893 DEBUG [Listener at localhost/46495] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 13:53:36,901 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39787 2023-04-12 13:53:36,902 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39787 2023-04-12 13:53:36,902 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39787 2023-04-12 13:53:36,902 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39787 2023-04-12 13:53:36,904 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39787 2023-04-12 13:53:36,910 INFO [Listener at localhost/46495] master.HMaster(444): hbase.rootdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964, hbase.cluster.distributed=false 2023-04-12 13:53:36,980 INFO [Listener at localhost/46495] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 13:53:36,980 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 13:53:36,980 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 13:53:36,980 INFO [Listener at localhost/46495] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 13:53:36,980 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 13:53:36,981 INFO [Listener at localhost/46495] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 13:53:36,986 INFO [Listener at localhost/46495] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 13:53:36,988 INFO [Listener at localhost/46495] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35663 2023-04-12 13:53:36,991 INFO [Listener at localhost/46495] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-12 13:53:36,998 DEBUG [Listener at localhost/46495] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-12 13:53:37,000 INFO [Listener at localhost/46495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:37,003 INFO [Listener at localhost/46495] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:37,004 INFO [Listener at localhost/46495] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35663 connecting to ZooKeeper ensemble=127.0.0.1:58527 2023-04-12 13:53:37,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:356630x0, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 13:53:37,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35663-0x10178a628840001 connected 2023-04-12 13:53:37,009 DEBUG [Listener at localhost/46495] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 13:53:37,011 DEBUG [Listener at localhost/46495] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 13:53:37,012 DEBUG [Listener at localhost/46495] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 13:53:37,012 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35663 2023-04-12 13:53:37,012 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35663 2023-04-12 13:53:37,016 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35663 2023-04-12 13:53:37,018 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35663 2023-04-12 13:53:37,019 DEBUG [Listener at localhost/46495] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35663 2023-04-12 13:53:37,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:37,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-12 13:53:37,035 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:37,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 13:53:37,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 13:53:37,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:37,056 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-12 13:53:37,057 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,39787,1681307615774 from backup master directory 2023-04-12 13:53:37,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-12 13:53:37,061 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:37,061 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-12 13:53:37,062 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 13:53:37,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:37,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-12 13:53:37,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-12 13:53:37,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/hbase.id with ID: 4cc862f5-dfd0-4fff-a1c8-bc8ae2f35dfb 2023-04-12 13:53:37,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 13:53:37,619 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:38,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x60f248a7 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:38,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a5ca16e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:38,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:53:38,140 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-12 13:53:38,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-12 13:53:38,161 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-12 13:53:38,163 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:53:38,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:53:38,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 13:53:38,205 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store-tmp 2023-04-12 13:53:38,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:38,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-12 13:53:38,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:53:38,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:53:38,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-12 13:53:38,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:53:38,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:53:38,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 13:53:38,240 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/WALs/jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:38,260 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39787%2C1681307615774, suffix=, logDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/WALs/jenkins-hbase4.apache.org,39787,1681307615774, archiveDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/oldWALs, maxLogs=10 2023-04-12 13:53:38,316 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38341,DS-33248ea2-084d-465e-ba85-b2841516c3d6,DISK] 2023-04-12 13:53:38,324 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:53:38,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/WALs/jenkins-hbase4.apache.org,39787,1681307615774/jenkins-hbase4.apache.org%2C39787%2C1681307615774.1681307618270 2023-04-12 13:53:38,391 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38341,DS-33248ea2-084d-465e-ba85-b2841516c3d6,DISK]] 2023-04-12 13:53:38,391 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:53:38,392 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:38,395 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 13:53:38,397 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 13:53:38,457 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-12 13:53:38,464 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-12 13:53:38,491 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-12 13:53:38,505 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:38,512 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-12 13:53:38,514 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-12 13:53:38,534 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 13:53:38,540 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:53:38,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66336129, jitterRate=-0.011514648795127869}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 13:53:38,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 13:53:38,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-12 13:53:38,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-12 13:53:38,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-12 13:53:38,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 13:53:38,574 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-12 13:53:38,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-04-12 13:53:38,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-12 13:53:38,643 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-12 13:53:38,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-12 13:53:38,678 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-12 13:53:38,682 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-12 13:53:38,685 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-12 13:53:38,690 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-12 13:53:38,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-12 13:53:38,699 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:38,701 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-12 13:53:38,702 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-12 13:53:38,716 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-12 13:53:38,721 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 13:53:38,721 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 13:53:38,721 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:38,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,39787,1681307615774, sessionid=0x10178a628840000, setting cluster-up flag (Was=false) 2023-04-12 13:53:38,742 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:38,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-12 13:53:38,750 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:38,757 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:38,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-12 13:53:38,765 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:38,769 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.hbase-snapshot/.tmp 2023-04-12 13:53:38,824 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(952): ClusterId : 4cc862f5-dfd0-4fff-a1c8-bc8ae2f35dfb 2023-04-12 13:53:38,831 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-12 13:53:38,840 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-12 13:53:38,840 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-12 13:53:38,844 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-12 13:53:38,845 DEBUG [RS:0;jenkins-hbase4:35663] zookeeper.ReadOnlyZKClient(139): Connect 0x79c8271d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:38,860 DEBUG [RS:0;jenkins-hbase4:35663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@770fc654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:38,861 DEBUG [RS:0;jenkins-hbase4:35663] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73304c6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 13:53:38,894 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35663 2023-04-12 13:53:38,900 INFO [RS:0;jenkins-hbase4:35663] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-12 13:53:38,901 INFO [RS:0;jenkins-hbase4:35663] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-12 13:53:38,901 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1023): About to register with Master. 2023-04-12 13:53:38,905 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,39787,1681307615774 with isa=jenkins-hbase4.apache.org/172.31.14.131:35663, startcode=1681307616979 2023-04-12 13:53:38,926 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-12 13:53:38,927 DEBUG [RS:0;jenkins-hbase4:35663] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-12 13:53:38,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 13:53:38,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 13:53:38,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 13:53:38,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 13:53:38,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-12 13:53:38,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:38,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 13:53:38,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:38,948 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681307648947 2023-04-12 13:53:38,950 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-12 13:53:38,952 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-12 13:53:38,952 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-12 13:53:38,959 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-12 13:53:38,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-12 13:53:38,973 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-12 13:53:38,973 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-12 13:53:38,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-12 13:53:38,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-12 13:53:38,975 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:38,977 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-12 13:53:38,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-12 13:53:38,979 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-12 13:53:38,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-12 13:53:38,982 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-12 13:53:38,984 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681307618984,5,FailOnTimeoutGroup] 2023-04-12 13:53:38,984 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681307618984,5,FailOnTimeoutGroup] 2023-04-12 13:53:38,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:38,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-12 13:53:38,987 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:38,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:38,990 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49441, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-12 13:53:38,993 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-12 13:53:38,995 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-12 13:53:38,995 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964 2023-04-12 13:53:39,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,008 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:39,011 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-12 13:53:39,013 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/info 2023-04-12 13:53:39,014 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-12 13:53:39,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:39,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-12 13:53:39,019 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/rep_barrier 2023-04-12 13:53:39,020 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-12 13:53:39,021 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964 2023-04-12 13:53:39,022 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:34129 2023-04-12 13:53:39,022 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-12 13:53:39,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:39,022 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-12 13:53:39,025 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/table 2023-04-12 13:53:39,026 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-12 13:53:39,027 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:39,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 13:53:39,029 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740 2023-04-12 13:53:39,029 DEBUG [RS:0;jenkins-hbase4:35663] zookeeper.ZKUtil(162): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,029 WARN [RS:0;jenkins-hbase4:35663] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 13:53:39,030 INFO [RS:0;jenkins-hbase4:35663] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 13:53:39,030 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1947): logDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/WALs/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,030 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740 2023-04-12 13:53:39,031 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35663,1681307616979] 2023-04-12 13:53:39,038 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:53:39,041 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-12 13:53:39,042 DEBUG [RS:0;jenkins-hbase4:35663] zookeeper.ZKUtil(162): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,049 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:53:39,055 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73780118, jitterRate=0.09940943121910095}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:53:39,055 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-12 13:53:39,055 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-12 13:53:39,055 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-12 13:53:39,055 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-12 13:53:39,056 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-12 13:53:39,056 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-12 13:53:39,058 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-12 13:53:39,059 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-12 13:53:39,059 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-12 13:53:39,067 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-12 13:53:39,067 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-12 13:53:39,069 INFO [RS:0;jenkins-hbase4:35663] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-12 13:53:39,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-12 13:53:39,095 INFO [RS:0;jenkins-hbase4:35663] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-12 13:53:39,101 INFO [RS:0;jenkins-hbase4:35663] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-12 13:53:39,101 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,103 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-12 13:53:39,104 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-12 13:53:39,107 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-12 13:53:39,117 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,117 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,117 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,118 DEBUG [RS:0;jenkins-hbase4:35663] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 13:53:39,120 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,120 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,120 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,142 INFO [RS:0;jenkins-hbase4:35663] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-12 13:53:39,144 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35663,1681307616979-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,162 INFO [RS:0;jenkins-hbase4:35663] regionserver.Replication(203): jenkins-hbase4.apache.org,35663,1681307616979 started 2023-04-12 13:53:39,162 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35663,1681307616979, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35663, sessionid=0x10178a628840001 2023-04-12 13:53:39,163 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-12 13:53:39,163 DEBUG [RS:0;jenkins-hbase4:35663] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,163 DEBUG [RS:0;jenkins-hbase4:35663] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35663,1681307616979' 2023-04-12 13:53:39,163 DEBUG [RS:0;jenkins-hbase4:35663] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:39,164 DEBUG [RS:0;jenkins-hbase4:35663] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:39,164 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-12 13:53:39,164 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-12 13:53:39,164 DEBUG [RS:0;jenkins-hbase4:35663] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,164 DEBUG [RS:0;jenkins-hbase4:35663] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35663,1681307616979' 2023-04-12 13:53:39,164 DEBUG [RS:0;jenkins-hbase4:35663] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-12 13:53:39,165 DEBUG [RS:0;jenkins-hbase4:35663] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-12 13:53:39,165 DEBUG [RS:0;jenkins-hbase4:35663] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-12 13:53:39,165 INFO [RS:0;jenkins-hbase4:35663] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-12 13:53:39,166 INFO [RS:0;jenkins-hbase4:35663] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-12 13:53:39,259 DEBUG [jenkins-hbase4:39787] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-12 13:53:39,262 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35663,1681307616979, state=OPENING 2023-04-12 13:53:39,269 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-12 13:53:39,271 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:39,271 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-12 13:53:39,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:53:39,280 INFO [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35663%2C1681307616979, suffix=, logDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/WALs/jenkins-hbase4.apache.org,35663,1681307616979, archiveDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/oldWALs, maxLogs=32 2023-04-12 13:53:39,304 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38341,DS-33248ea2-084d-465e-ba85-b2841516c3d6,DISK] 2023-04-12 13:53:39,308 INFO [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/WALs/jenkins-hbase4.apache.org,35663,1681307616979/jenkins-hbase4.apache.org%2C35663%2C1681307616979.1681307619285 2023-04-12 13:53:39,309 DEBUG [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38341,DS-33248ea2-084d-465e-ba85-b2841516c3d6,DISK]] 2023-04-12 13:53:39,466 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:39,468 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-12 13:53:39,472 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-12 13:53:39,484 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-12 13:53:39,485 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 13:53:39,489 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35663%2C1681307616979.meta, suffix=.meta, logDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/WALs/jenkins-hbase4.apache.org,35663,1681307616979, archiveDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/oldWALs, maxLogs=32 2023-04-12 13:53:39,508 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38341,DS-33248ea2-084d-465e-ba85-b2841516c3d6,DISK] 2023-04-12 13:53:39,513 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/WALs/jenkins-hbase4.apache.org,35663,1681307616979/jenkins-hbase4.apache.org%2C35663%2C1681307616979.meta.1681307619490.meta 2023-04-12 13:53:39,513 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38341,DS-33248ea2-084d-465e-ba85-b2841516c3d6,DISK]] 2023-04-12 13:53:39,513 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:53:39,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-12 13:53:39,536 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-12 13:53:39,541 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-12 13:53:39,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-12 13:53:39,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:39,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-12 13:53:39,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-12 13:53:39,551 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-12 13:53:39,552 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/info 2023-04-12 13:53:39,553 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/info 2023-04-12 13:53:39,553 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-12 13:53:39,554 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:39,554 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-12 13:53:39,556 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/rep_barrier 2023-04-12 13:53:39,556 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/rep_barrier 2023-04-12 13:53:39,557 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-12 13:53:39,558 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:39,558 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-12 13:53:39,560 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/table 2023-04-12 13:53:39,560 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/table 2023-04-12 13:53:39,560 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-12 13:53:39,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:39,563 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740 2023-04-12 13:53:39,566 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740 2023-04-12 13:53:39,570 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:53:39,572 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-12 13:53:39,574 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73937203, jitterRate=0.10175018012523651}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:53:39,575 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-12 13:53:39,584 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681307619456 2023-04-12 13:53:39,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-12 13:53:39,603 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-12 13:53:39,604 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35663,1681307616979, state=OPEN 2023-04-12 13:53:39,607 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-12 13:53:39,607 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-12 13:53:39,611 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-12 13:53:39,611 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35663,1681307616979 in 333 msec 2023-04-12 13:53:39,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-12 13:53:39,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 534 msec 2023-04-12 13:53:39,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 795 msec 2023-04-12 13:53:39,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681307619622, completionTime=-1 2023-04-12 13:53:39,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-12 13:53:39,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 13:53:39,689 DEBUG [hconnection-0x7de1e028-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:39,693 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:39,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-12 13:53:39,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681307679720 2023-04-12 13:53:39,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681307739720 2023-04-12 13:53:39,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 96 msec 2023-04-12 13:53:39,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39787,1681307615774-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39787,1681307615774-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39787,1681307615774-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:39787, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-12 13:53:39,754 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-12 13:53:39,765 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-12 13:53:39,768 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-12 13:53:39,779 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-12 13:53:39,782 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:53:39,785 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:53:39,808 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:39,810 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb empty. 2023-04-12 13:53:39,811 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:39,811 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-12 13:53:40,233 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-12 13:53:40,236 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 083bda99a45a5ebe3c6fdff5c36223cb, NAME => 'hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:53:40,248 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:40,249 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 083bda99a45a5ebe3c6fdff5c36223cb, disabling compactions & flushes 2023-04-12 13:53:40,249 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,249 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,249 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. after waiting 0 ms 2023-04-12 13:53:40,249 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,249 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,249 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 083bda99a45a5ebe3c6fdff5c36223cb: 2023-04-12 13:53:40,254 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:53:40,271 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681307620257"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307620257"}]},"ts":"1681307620257"} 2023-04-12 13:53:40,306 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:53:40,308 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:53:40,314 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307620309"}]},"ts":"1681307620309"} 2023-04-12 13:53:40,320 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-12 13:53:40,330 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=083bda99a45a5ebe3c6fdff5c36223cb, ASSIGN}] 2023-04-12 13:53:40,333 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=083bda99a45a5ebe3c6fdff5c36223cb, ASSIGN 2023-04-12 13:53:40,335 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=083bda99a45a5ebe3c6fdff5c36223cb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:53:40,486 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=083bda99a45a5ebe3c6fdff5c36223cb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:40,487 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681307620486"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307620486"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307620486"}]},"ts":"1681307620486"} 2023-04-12 13:53:40,491 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 083bda99a45a5ebe3c6fdff5c36223cb, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:53:40,652 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,653 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 083bda99a45a5ebe3c6fdff5c36223cb, NAME => 'hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:53:40,654 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,654 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:40,654 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,654 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,656 INFO [StoreOpener-083bda99a45a5ebe3c6fdff5c36223cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,659 DEBUG [StoreOpener-083bda99a45a5ebe3c6fdff5c36223cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/info 2023-04-12 13:53:40,659 DEBUG [StoreOpener-083bda99a45a5ebe3c6fdff5c36223cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/info 2023-04-12 13:53:40,659 INFO [StoreOpener-083bda99a45a5ebe3c6fdff5c36223cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 083bda99a45a5ebe3c6fdff5c36223cb columnFamilyName info 2023-04-12 13:53:40,660 INFO [StoreOpener-083bda99a45a5ebe3c6fdff5c36223cb-1] regionserver.HStore(310): Store=083bda99a45a5ebe3c6fdff5c36223cb/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:40,662 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,663 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,667 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:53:40,670 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:53:40,671 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 083bda99a45a5ebe3c6fdff5c36223cb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60412555, jitterRate=-0.0997827798128128}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 13:53:40,671 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 083bda99a45a5ebe3c6fdff5c36223cb: 2023-04-12 13:53:40,673 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb., pid=6, masterSystemTime=1681307620646 2023-04-12 13:53:40,678 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,678 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:53:40,679 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=083bda99a45a5ebe3c6fdff5c36223cb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:40,680 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681307620678"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307620678"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307620678"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307620678"}]},"ts":"1681307620678"} 2023-04-12 13:53:40,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-12 13:53:40,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 083bda99a45a5ebe3c6fdff5c36223cb, server=jenkins-hbase4.apache.org,35663,1681307616979 in 194 msec 2023-04-12 13:53:40,692 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-12 13:53:40,693 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=083bda99a45a5ebe3c6fdff5c36223cb, ASSIGN in 359 msec 2023-04-12 13:53:40,694 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:53:40,694 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307620694"}]},"ts":"1681307620694"} 2023-04-12 13:53:40,697 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-12 13:53:40,701 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:53:40,703 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 933 msec 2023-04-12 13:53:40,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-12 13:53:40,784 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-12 13:53:40,784 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:53:40,822 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-12 13:53:40,842 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-12 13:53:40,848 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-12 13:53:40,856 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-12 13:53:40,867 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-12 13:53:40,872 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-04-12 13:53:40,883 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-12 13:53:40,886 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-12 13:53:40,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.824sec 2023-04-12 13:53:40,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-12 13:53:40,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-12 13:53:40,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-12 13:53:40,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39787,1681307615774-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-12 13:53:40,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39787,1681307615774-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-12 13:53:40,908 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-12 13:53:40,931 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:40,938 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:40,954 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:40,966 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:40,977 INFO [Listener at localhost/46495] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:53:40,999 INFO [Listener at localhost/46495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=355, ProcessCount=169, AvailableMemoryMB=1221 2023-04-12 13:53:41,018 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:53:41,055 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:53:41,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:53:41,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:53:41,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 13:53:41,073 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:53:41,076 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:53:41,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-12 13:53:41,079 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,081 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f empty. 2023-04-12 13:53:41,083 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,083 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:53:41,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 13:53:41,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 13:53:41,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 13:53:41,504 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:53:41,506 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 61f538e965579a8c5df754be9b63fe6f, NAME => 'TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:53:41,522 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:41,523 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 61f538e965579a8c5df754be9b63fe6f, disabling compactions & flushes 2023-04-12 13:53:41,523 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,523 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,523 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. after waiting 0 ms 2023-04-12 13:53:41,523 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,523 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,523 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:41,528 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:53:41,529 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307621529"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307621529"}]},"ts":"1681307621529"} 2023-04-12 13:53:41,532 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:53:41,534 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:53:41,534 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307621534"}]},"ts":"1681307621534"} 2023-04-12 13:53:41,537 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 13:53:41,541 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, ASSIGN}] 2023-04-12 13:53:41,544 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, ASSIGN 2023-04-12 13:53:41,547 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:53:41,698 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=61f538e965579a8c5df754be9b63fe6f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:41,699 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307621698"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307621698"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307621698"}]},"ts":"1681307621698"} 2023-04-12 13:53:41,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 13:53:41,702 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:53:41,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 61f538e965579a8c5df754be9b63fe6f, NAME => 'TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:53:41,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,862 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:53:41,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,864 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,866 DEBUG [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A 2023-04-12 13:53:41,867 DEBUG [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A 2023-04-12 13:53:41,868 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:53:41,868 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61f538e965579a8c5df754be9b63fe6f columnFamilyName A 2023-04-12 13:53:41,869 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.HStore(310): Store=61f538e965579a8c5df754be9b63fe6f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:41,869 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,871 DEBUG [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B 2023-04-12 13:53:41,871 DEBUG [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B 2023-04-12 13:53:41,872 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:53:41,872 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61f538e965579a8c5df754be9b63fe6f columnFamilyName B 2023-04-12 13:53:41,873 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.HStore(310): Store=61f538e965579a8c5df754be9b63fe6f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:41,873 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,875 DEBUG [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C 2023-04-12 13:53:41,875 DEBUG [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C 2023-04-12 13:53:41,875 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:53:41,875 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61f538e965579a8c5df754be9b63fe6f columnFamilyName C 2023-04-12 13:53:41,876 INFO [StoreOpener-61f538e965579a8c5df754be9b63fe6f-1] regionserver.HStore(310): Store=61f538e965579a8c5df754be9b63fe6f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:53:41,877 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:53:41,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:41,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:53:41,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 61f538e965579a8c5df754be9b63fe6f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65772651, jitterRate=-0.01991112530231476}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:53:41,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:41,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., pid=11, masterSystemTime=1681307621855 2023-04-12 13:53:41,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:41,894 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=61f538e965579a8c5df754be9b63fe6f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:41,894 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307621894"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307621894"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307621894"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307621894"}]},"ts":"1681307621894"} 2023-04-12 13:53:41,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-12 13:53:41,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 in 195 msec 2023-04-12 13:53:41,903 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-12 13:53:41,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, ASSIGN in 359 msec 2023-04-12 13:53:41,904 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:53:41,905 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307621905"}]},"ts":"1681307621905"} 2023-04-12 13:53:41,907 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 13:53:41,910 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:53:41,912 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 841 msec 2023-04-12 13:53:42,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-12 13:53:42,203 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-12 13:53:42,208 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,225 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,229 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,232 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,240 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:53:42,243 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:53:42,252 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,257 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,258 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,263 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,263 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,269 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,270 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,276 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,277 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,284 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,285 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,291 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,291 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,328 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,329 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,335 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,336 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:53:42,339 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:53:42,345 DEBUG [hconnection-0x2e76d9bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,346 DEBUG [hconnection-0x224073f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,346 DEBUG [hconnection-0x3781157c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,349 DEBUG [hconnection-0x417c8b83-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,349 DEBUG [hconnection-0x9312249-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,349 DEBUG [hconnection-0x3df3bc44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,350 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,352 DEBUG [hconnection-0x5ab73287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,352 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,355 DEBUG [hconnection-0x217a24de-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,357 DEBUG [hconnection-0x74c2cd1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:53:42,358 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55668, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,358 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,359 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,369 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,369 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55682, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,378 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55696, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:53:42,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:42,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:42,395 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:42,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:42,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:42,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:42,396 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,396 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:42,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,400 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:42,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:42,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:42,402 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,402 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:42,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:42,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:42,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:42,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:42,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:42,416 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:42,419 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:42,422 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-12 13:53:42,442 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:42,443 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:42,444 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:42,444 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:42,444 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:42,444 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:42,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:42,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:42,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307622523, max timestamp=1681307622557Num uniques 859; 2023-04-12 13:53:42,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307622523, max timestamp=1681307622557Num uniques 859; 2023-04-12 13:53:42,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:53:42,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-12 13:53:42,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307622523, max timestamp=1681307622557Num uniques 859; 2023-04-12 13:53:42,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:53:42,615 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/65e87fc79a524789b08d02cbb4cbf5ba 2023-04-12 13:53:42,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307682651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307682655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307682656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307682658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307682661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,762 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/8d7e6bfd06ee4e58b09eeb68197fb63b 2023-04-12 13:53:42,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307682792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307682792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307682795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307682796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307682796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,837 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/c1f32eacd9204ba88b1fe2f8d1282ad1 2023-04-12 13:53:42,854 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/65e87fc79a524789b08d02cbb4cbf5ba as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/65e87fc79a524789b08d02cbb4cbf5ba 2023-04-12 13:53:42,869 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/65e87fc79a524789b08d02cbb4cbf5ba, entries=150, sequenceid=10, filesize=11.6 K 2023-04-12 13:53:42,882 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/8d7e6bfd06ee4e58b09eeb68197fb63b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/8d7e6bfd06ee4e58b09eeb68197fb63b 2023-04-12 13:53:42,904 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/8d7e6bfd06ee4e58b09eeb68197fb63b, entries=150, sequenceid=10, filesize=11.6 K 2023-04-12 13:53:42,907 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/c1f32eacd9204ba88b1fe2f8d1282ad1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c1f32eacd9204ba88b1fe2f8d1282ad1 2023-04-12 13:53:42,919 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c1f32eacd9204ba88b1fe2f8d1282ad1, entries=150, sequenceid=10, filesize=11.6 K 2023-04-12 13:53:42,922 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=138.26 KB/141575 for 61f538e965579a8c5df754be9b63fe6f in 500ms, sequenceid=10, compaction requested=false 2023-04-12 13:53:42,924 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 13:53:42,928 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:42,928 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:42,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:42,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:42,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:42,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:42,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:42,936 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,936 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:42,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:42,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c9a2a2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:42,939 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:42,939 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,943 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,943 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:42,943 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:42,944 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:42,944 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:42,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:42,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:42,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:42,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:42,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:42,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:42,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:42,960 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:42,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:42,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:42,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:42,960 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:42,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:42,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:42,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:42,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:42,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:42,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:42,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:42,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:42,996 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:42,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:42,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:42,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:42,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:42,998 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:42,998 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:42,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:43,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=144.97 KB heapSize=414.85 KB 2023-04-12 13:53:43,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:43,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:53:43,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:43,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:53:43,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:43,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:53:43,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307683042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307683045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307683048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307683051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307683055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,099 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:43,119 WARN [DataStreamer for file /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/8c9b7cf434574d448ce9dd2ba0297aec] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-12 13:53:43,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/8c9b7cf434574d448ce9dd2ba0297aec 2023-04-12 13:53:43,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:43,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:43,139 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:43,139 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:43,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:43,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,144 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,144 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,146 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,146 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:43,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:43,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:43,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:43,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:43,165 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,165 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:43,165 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:43,165 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:43,171 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:43,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5284adbd[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:43,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:43,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:43,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:43,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:43,189 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:43,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307683161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307683167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307683173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307683174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:43,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:43,219 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:43,219 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:43,219 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:43,219 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:43,219 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:43,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307683177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,230 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,230 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,231 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,234 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/a12a4ef1f40946f0986d26cf780a261b 2023-04-12 13:53:43,246 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,246 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,246 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,246 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,246 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,252 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,252 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,252 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,253 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:43,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:43,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:43,253 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:43,253 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:43,353 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:43,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:43,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:43,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:43,392 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:43,392 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,393 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:43,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:43,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,399 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:43,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:43,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:43,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:43,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:43,413 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,413 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:43,413 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:43,413 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:43,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:43,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:43,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8a018d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:43,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:43,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307683415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:43,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:43,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:43,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307683416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307683416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:43,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:43,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:43,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307683421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307683433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:43,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:43,458 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:43,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:43,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:43,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:43,458 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:43,461 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,461 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:43,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,461 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,463 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,465 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,487 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:43,487 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:43,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:43,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:43,487 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,487 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:43,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,494 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:43,494 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:43,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,594 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:43,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:43,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:43,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:43,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:43,624 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:43,624 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:43,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:43,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,648 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:43,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:43,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:43,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:43,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:43,657 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,657 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:43,657 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:43,658 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:43,658 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:43,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:43,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:43,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:43,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:43,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:43,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55fcd3ca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:43,665 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:43,665 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:43,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:43,669 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:43,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:43,671 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:43,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:43,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:43,688 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:43,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:43,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:43,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:43,688 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:43,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:43,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:43,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:43,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:43,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:43,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:43,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:43,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:43,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:43,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,714 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:43,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:43,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/acffe2a0705346fdacf53f988b4a5b26 2023-04-12 13:53:43,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/8c9b7cf434574d448ce9dd2ba0297aec as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8c9b7cf434574d448ce9dd2ba0297aec 2023-04-12 13:53:43,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307683738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8c9b7cf434574d448ce9dd2ba0297aec, entries=200, sequenceid=48, filesize=13.9 K 2023-04-12 13:53:43,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307683743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307683743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:43,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307683746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/a12a4ef1f40946f0986d26cf780a261b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a12a4ef1f40946f0986d26cf780a261b 2023-04-12 13:53:43,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307683744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a12a4ef1f40946f0986d26cf780a261b, entries=150, sequenceid=48, filesize=11.6 K 2023-04-12 13:53:43,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/acffe2a0705346fdacf53f988b4a5b26 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/acffe2a0705346fdacf53f988b4a5b26 2023-04-12 13:53:43,814 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/acffe2a0705346fdacf53f988b4a5b26, entries=150, sequenceid=48, filesize=11.6 K 2023-04-12 13:53:43,814 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:43,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:43,823 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~144.97 KB/148445, heapSize ~414.80 KB/424760, currentSize=33.54 KB/34350 for 61f538e965579a8c5df754be9b63fe6f in 786ms, sequenceid=48, compaction requested=false 2023-04-12 13:53:43,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:43,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:43,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:43,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:43,837 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:43,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:43,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:43,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:43,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:43,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:43,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:43,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:43,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:43,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:43,856 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:43,856 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:43,857 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-12 13:53:43,857 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:43,858 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:43,858 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:43,858 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:43,859 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:43,859 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:43,968 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/986cc88899c34444a1e890b8cc755cd1 2023-04-12 13:53:44,056 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/00401b40776d473ead34e0a2c071b991 2023-04-12 13:53:44,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:44,261 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:44,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307624289, max timestamp=1681307624297Num uniques 859; 2023-04-12 13:53:44,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:53:44,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307624289, max timestamp=1681307624294Num uniques 859; 2023-04-12 13:53:44,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-12 13:53:44,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307684334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307684335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307684335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307684338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307684338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307684441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307684442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307684442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307684448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307684449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,538 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=56 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/f06bbe90e10143f09d48de0f8b396dbd 2023-04-12 13:53:44,557 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/986cc88899c34444a1e890b8cc755cd1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/986cc88899c34444a1e890b8cc755cd1 2023-04-12 13:53:44,583 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/986cc88899c34444a1e890b8cc755cd1, entries=150, sequenceid=56, filesize=11.6 K 2023-04-12 13:53:44,589 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/00401b40776d473ead34e0a2c071b991 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/00401b40776d473ead34e0a2c071b991 2023-04-12 13:53:44,629 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/00401b40776d473ead34e0a2c071b991, entries=150, sequenceid=56, filesize=11.6 K 2023-04-12 13:53:44,634 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/f06bbe90e10143f09d48de0f8b396dbd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/f06bbe90e10143f09d48de0f8b396dbd 2023-04-12 13:53:44,649 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/f06bbe90e10143f09d48de0f8b396dbd, entries=150, sequenceid=56, filesize=11.6 K 2023-04-12 13:53:44,651 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=144.97 KB/148445 for 61f538e965579a8c5df754be9b63fe6f in 794ms, sequenceid=56, compaction requested=true 2023-04-12 13:53:44,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:44,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:44,651 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:44,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:44,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97ca765[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:44,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:44,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,663 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:44,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:44,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:44,664 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,664 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:44,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:44,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:44,670 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:44,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:44,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:44,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:44,670 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:44,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:44,673 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,673 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=151.67 KB heapSize=442.16 KB 2023-04-12 13:53:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:44,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:44,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:44,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:44,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:44,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:44,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:44,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:44,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:44,695 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:44,695 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:44,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:44,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307684690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307684692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307684703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307684703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307684708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,798 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:44,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:44,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307684816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:44,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307684822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307684830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307684830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:44,836 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:44,836 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:44,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:44,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:44,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:44,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:44,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:44,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:44,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:44,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:44,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:44,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:44,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:44,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:44,851 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:44,851 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:44,851 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:44,852 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:44,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:44,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee21ab2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:44,858 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:44,858 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:44,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:44,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:44,868 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:44,868 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:44,868 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:44,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307684856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:44,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:44,874 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:44,875 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:44,875 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:44,875 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:44,875 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:44,879 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,879 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:44,879 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:44,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,884 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:44,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:44,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:44,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:44,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:44,887 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:44,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:44,893 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:44,891 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:44,894 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:44,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:44,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:44,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:44,994 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:44,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:44,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:45,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:45,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:45,013 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:45,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:45,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:45,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,017 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:45,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:45,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:45,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,031 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:45,044 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:45,044 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307685037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307685048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307685048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:45,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307685048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:45,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,063 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:45,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756fea8a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:45,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:45,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:45,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:45,068 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:45,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:45,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:45,079 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:45,079 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:45,080 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:45,080 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:45,080 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:45,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,083 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,083 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,087 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307685084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,101 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,101 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,102 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,102 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,102 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:45,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:45,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:45,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,103 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:45,103 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:45,143 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 13:53:45,203 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:45,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/2331f2d93b304899be5c851d343cf6bc 2023-04-12 13:53:45,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:45,255 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:45,255 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,262 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:45,262 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:45,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,265 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:45,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:45,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:45,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:45,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:45,293 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,293 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:45,293 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:45,293 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:45,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:45,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463e156e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:45,338 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:45,338 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,340 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:45,340 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:45,340 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,340 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:45,345 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:45,345 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,345 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,345 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:45,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:45,356 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:45,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:45,356 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:45,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307685358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307685362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307685363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,364 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,372 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:45,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,373 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,373 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,373 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:45,374 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:45,374 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,374 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307685365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:45,393 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-12 13:53:45,396 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-12 13:53:45,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55702 deadline: 1681307685392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,474 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:45,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:45,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:45,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:45,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:45,492 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:45,492 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,494 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:45,494 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:45,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,497 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,498 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:45,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:45,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:45,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:45,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:45,500 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,505 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:45,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:45,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:45,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:45,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:45,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:45,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,511 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:45,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a15c543[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:45,515 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:45,515 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,516 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:45,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:45,518 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:45,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:45,523 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:45,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:45,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:45,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:45,523 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:45,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,527 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,533 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,533 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:45,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:45,533 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,534 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:45,534 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:45,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:45,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:45,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:45,648 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:45,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:45,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:45,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,655 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,655 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,656 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,656 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:45,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:45,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:45,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:45,676 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:45,676 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:45,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:45,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44cc6d39[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:45,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:45,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:45,698 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:45,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:45,714 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:45,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:45,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:45,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:45,714 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:45,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,717 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:45,720 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/630cb0368cd84417b7c4211c896bf14e 2023-04-12 13:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,726 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,726 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:45,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:45,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:45,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:45,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:45,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:45,729 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:45,730 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:45,776 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=73.80 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/7f3daffd2da14bf0901e1da3f7a90589 2023-04-12 13:53:45,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/2331f2d93b304899be5c851d343cf6bc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2331f2d93b304899be5c851d343cf6bc 2023-04-12 13:53:45,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2331f2d93b304899be5c851d343cf6bc, entries=150, sequenceid=92, filesize=11.6 K 2023-04-12 13:53:45,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/630cb0368cd84417b7c4211c896bf14e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/630cb0368cd84417b7c4211c896bf14e 2023-04-12 13:53:45,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/630cb0368cd84417b7c4211c896bf14e, entries=150, sequenceid=92, filesize=11.6 K 2023-04-12 13:53:45,830 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:45,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:45,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/7f3daffd2da14bf0901e1da3f7a90589 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7f3daffd2da14bf0901e1da3f7a90589 2023-04-12 13:53:45,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307685865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307685867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:45,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:45,870 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:45,870 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307685868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:45,873 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:45,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307685886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:45,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,917 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:45,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:45,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:45,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7f3daffd2da14bf0901e1da3f7a90589, entries=150, sequenceid=92, filesize=11.6 K 2023-04-12 13:53:45,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:45,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~158.38 KB/162185, heapSize ~462.04 KB/473128, currentSize=20.13 KB/20610 for 61f538e965579a8c5df754be9b63fe6f in 1233ms, sequenceid=92, compaction requested=true 2023-04-12 13:53:45,922 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:45,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:45,930 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-12 13:53:45,931 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-12 13:53:45,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:45,969 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:45,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:45,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:45,970 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 13:53:45,971 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:53:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:45,975 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:53:45,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:45,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:45,988 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,988 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:45,988 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 13:53:45,994 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:53:45,997 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:53:45,997 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:45,997 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:45,997 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,997 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:45,997 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/8d7e6bfd06ee4e58b09eeb68197fb63b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a12a4ef1f40946f0986d26cf780a261b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/00401b40776d473ead34e0a2c071b991, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/630cb0368cd84417b7c4211c896bf14e] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=46.4 K 2023-04-12 13:53:45,997 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/65e87fc79a524789b08d02cbb4cbf5ba, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8c9b7cf434574d448ce9dd2ba0297aec, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/986cc88899c34444a1e890b8cc755cd1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2331f2d93b304899be5c851d343cf6bc] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=48.7 K 2023-04-12 13:53:46,002 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 65e87fc79a524789b08d02cbb4cbf5ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681307622380 2023-04-12 13:53:46,003 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 8c9b7cf434574d448ce9dd2ba0297aec, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681307622651 2023-04-12 13:53:46,003 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 986cc88899c34444a1e890b8cc755cd1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681307623044 2023-04-12 13:53:46,004 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8d7e6bfd06ee4e58b09eeb68197fb63b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681307622380 2023-04-12 13:53:46,007 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2331f2d93b304899be5c851d343cf6bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681307624333 2023-04-12 13:53:46,014 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a12a4ef1f40946f0986d26cf780a261b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681307622651 2023-04-12 13:53:46,015 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 00401b40776d473ead34e0a2c071b991, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681307623044 2023-04-12 13:53:46,016 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 630cb0368cd84417b7c4211c896bf14e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681307624333 2023-04-12 13:53:46,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:46,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:46,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:46,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:46,037 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:46,037 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:46,069 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#13 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:46,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:46,075 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#14 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:46,177 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/caa9582f7aa0431fb7d70e38505684a7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/caa9582f7aa0431fb7d70e38505684a7 2023-04-12 13:53:46,226 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into caa9582f7aa0431fb7d70e38505684a7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:46,227 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:46,227 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=12, startTime=1681307625924; duration=0sec 2023-04-12 13:53:46,228 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:46,228 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:53:46,248 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:53:46,249 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:46,249 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:46,249 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c1f32eacd9204ba88b1fe2f8d1282ad1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/acffe2a0705346fdacf53f988b4a5b26, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/f06bbe90e10143f09d48de0f8b396dbd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7f3daffd2da14bf0901e1da3f7a90589] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=46.4 K 2023-04-12 13:53:46,250 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c1f32eacd9204ba88b1fe2f8d1282ad1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1681307622380 2023-04-12 13:53:46,250 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting acffe2a0705346fdacf53f988b4a5b26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681307622651 2023-04-12 13:53:46,251 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f06bbe90e10143f09d48de0f8b396dbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=56, earliestPutTs=1681307623044 2023-04-12 13:53:46,252 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 7f3daffd2da14bf0901e1da3f7a90589, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681307624333 2023-04-12 13:53:46,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307626192, max timestamp=1681307626260Num uniques 859; 2023-04-12 13:53:46,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307626192, max timestamp=1681307626260Num uniques 859; 2023-04-12 13:53:46,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:53:46,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:53:46,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307626192, max timestamp=1681307626260Num uniques 859; 2023-04-12 13:53:46,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:53:46,305 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#15 average throughput is 0.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:46,358 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/48ddb73826094bb3bc50ee9c387e0f70 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/48ddb73826094bb3bc50ee9c387e0f70 2023-04-12 13:53:46,410 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 48ddb73826094bb3bc50ee9c387e0f70(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:46,410 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:46,410 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=12, startTime=1681307625970; duration=0sec 2023-04-12 13:53:46,410 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:46,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307626400, max timestamp=1681307626452Num uniques 1018; 2023-04-12 13:53:46,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 13:53:46,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307626400, max timestamp=1681307626452Num uniques 1018; 2023-04-12 13:53:46,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 13:53:46,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307626400, max timestamp=1681307626452Num uniques 1018; 2023-04-12 13:53:46,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 13:53:46,542 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/b87b8eab854b440fb32d06e7c0261021 2023-04-12 13:53:46,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681307626507, max timestamp=1681307626562Num uniques 1077; 2023-04-12 13:53:46,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-12 13:53:46,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681307626507, max timestamp=1681307626562Num uniques 1077; 2023-04-12 13:53:46,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-12 13:53:46,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681307626507, max timestamp=1681307626562Num uniques 1077; 2023-04-12 13:53:46,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-12 13:53:46,606 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/9f07fa1404f749a08e6dfec25a2d8839 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9f07fa1404f749a08e6dfec25a2d8839 2023-04-12 13:53:46,619 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into 9f07fa1404f749a08e6dfec25a2d8839(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:46,619 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:46,619 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=12, startTime=1681307625970; duration=0sec 2023-04-12 13:53:46,620 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681307626607, max timestamp=1681307626659Num uniques 1036; 2023-04-12 13:53:46,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-12 13:53:46,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681307626607, max timestamp=1681307626659Num uniques 1036; 2023-04-12 13:53:46,664 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-12 13:53:46,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681307626607, max timestamp=1681307626659Num uniques 1036; 2023-04-12 13:53:46,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-12 13:53:46,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681307626720, max timestamp=1681307626744Num uniques 1095; 2023-04-12 13:53:46,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681307626720, max timestamp=1681307626744Num uniques 1095; 2023-04-12 13:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681307626720, max timestamp=1681307626744Num uniques 1095; 2023-04-12 13:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-12 13:53:46,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-12 13:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-12 13:53:46,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307626795, max timestamp=1681307626831Num uniques 1054; 2023-04-12 13:53:46,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307626795, max timestamp=1681307626831Num uniques 1054; 2023-04-12 13:53:46,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307626795, max timestamp=1681307626831Num uniques 1054; 2023-04-12 13:53:46,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 13:53:46,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 13:53:46,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 13:53:46,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1681307626881, max timestamp=1681307626896Num uniques 1013; 2023-04-12 13:53:46,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-04-12 13:53:46,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1681307626881, max timestamp=1681307626896Num uniques 1013; 2023-04-12 13:53:46,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=16618, suffix heap size=133320,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=363 2023-04-12 13:53:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307626887, max timestamp=1681307626896Num uniques 1013; 2023-04-12 13:53:46,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-12 13:53:46,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:46,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307686923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:46,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:46,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307686924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:46,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:46,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307686924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:46,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:46,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307686925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:46,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:46,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307686926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:46,996 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/37fe4113fde7455f858baa0522b66c99 2023-04-12 13:53:47,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307687031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307687033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307687033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307687037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307687034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307687236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307687241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307687241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307687245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307687246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,435 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/28516c23bb3a41dca56a26d87d7a984f 2023-04-12 13:53:47,446 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/b87b8eab854b440fb32d06e7c0261021 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b87b8eab854b440fb32d06e7c0261021 2023-04-12 13:53:47,457 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b87b8eab854b440fb32d06e7c0261021, entries=150, sequenceid=100, filesize=11.6 K 2023-04-12 13:53:47,459 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/37fe4113fde7455f858baa0522b66c99 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/37fe4113fde7455f858baa0522b66c99 2023-04-12 13:53:47,470 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/37fe4113fde7455f858baa0522b66c99, entries=150, sequenceid=100, filesize=11.6 K 2023-04-12 13:53:47,475 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/28516c23bb3a41dca56a26d87d7a984f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/28516c23bb3a41dca56a26d87d7a984f 2023-04-12 13:53:47,503 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/28516c23bb3a41dca56a26d87d7a984f, entries=150, sequenceid=100, filesize=11.6 K 2023-04-12 13:53:47,506 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=152.07 KB/155720 for 61f538e965579a8c5df754be9b63fe6f in 1517ms, sequenceid=100, compaction requested=false 2023-04-12 13:53:47,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:47,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:47,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:47,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:47,512 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:47,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7de490[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:47,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:47,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,518 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:47,518 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,518 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:47,520 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:47,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:47,526 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:47,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:47,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:47,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:47,527 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:47,529 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,529 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:47,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,529 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:47,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,546 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:47,546 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:47,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:47,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681307626910, max timestamp=1681307627543Num uniques 1222; 2023-04-12 13:53:47,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=7872, suffix heap size=155056,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=172 2023-04-12 13:53:47,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681307626910, max timestamp=1681307627543Num uniques 1222; 2023-04-12 13:53:47,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=7872, suffix heap size=155056,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=172 2023-04-12 13:53:47,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=64.85 KB heapSize=178.42 KB 2023-04-12 13:53:47,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:47,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:47,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:47,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:47,550 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:47,550 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:53:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307627566, max timestamp=1681307627578Num uniques 859; 2023-04-12 13:53:47,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307687574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:53:47,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307627566, max timestamp=1681307627578Num uniques 859; 2023-04-12 13:53:47,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:53:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307627566, max timestamp=1681307627578Num uniques 859; 2023-04-12 13:53:47,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:53:47,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/d77cbe5295f6431dab1e637ea8a4d23e 2023-04-12 13:53:47,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307687632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307687632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307687635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307687636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,647 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:47,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:47,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:47,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:47,661 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:47,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:47,662 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:47,662 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:47,663 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,663 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:47,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:47,669 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:47,673 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,674 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:47,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78de0d54[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:47,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:47,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:47,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:47,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307687691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:47,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:47,700 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:47,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:47,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:47,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:47,700 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:47,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,712 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:47,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:47,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:47,716 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:47,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:47,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:47,717 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:47,718 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:47,718 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:47,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307687740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307687740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307687741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307687741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,818 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:47,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:47,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:47,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:47,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:47,826 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:47,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:47,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:47,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:47,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,829 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:47,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:47,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:47,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:47,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:47,833 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,833 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:47,833 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:47,833 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,836 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:47,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce5f0d1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:47,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:47,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:47,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:47,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:47,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:47,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:47,851 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:47,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:47,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:47,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:47,851 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:47,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:47,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:47,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,857 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:47,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:47,862 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:47,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:47,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:47,862 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:47,864 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:47,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:47,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:47,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307687900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307687946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307687946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307687947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:47,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307687947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,964 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:47,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:47,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:47,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:47,976 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:47,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:47,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:47,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:47,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:47,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:47,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:47,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:47,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:47,984 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:47,986 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,986 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:47,986 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:47,986 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,988 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:47,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:47,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b67efd1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:47,991 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:47,991 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:47,994 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:47,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:47,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:48,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:48,002 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:48,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:48,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:48,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:48,002 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:48,005 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,005 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,005 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,006 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:48,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:48,017 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:48,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:48,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:48,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/0be9ad9f1db8491cb3ffdaafa6e4a772 2023-04-12 13:53:48,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:48,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:48,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:48,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:48,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:48,127 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:48,127 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:48,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:48,129 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,129 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,132 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,133 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,133 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:48,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:48,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:48,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:48,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:48,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:48,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:48,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:48,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:48,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:48,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:48,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1fc949[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:48,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:48,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:48,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:48,162 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,162 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:48,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:48,169 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:48,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:48,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:48,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:48,169 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:48,171 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,171 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,171 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,180 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,180 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:48,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:48,185 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:48,185 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:48,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307688204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307688252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307688252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307688252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307688255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,286 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:48,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:48,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:48,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:48,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:48,300 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:48,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:48,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:48,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,313 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:48,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:48,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:48,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:48,318 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:48,318 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:48,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:48,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:48,320 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:48,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:48,322 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,322 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,323 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,323 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72a3cff3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:48,326 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:48,327 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:48,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:48,331 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:48,331 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,331 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:48,332 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:48,347 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:48,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:48,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:48,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:48,347 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:48,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,351 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,353 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,356 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,356 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:48,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:48,361 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:48,361 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:48,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:48,461 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:48,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:48,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:48,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:48,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:48,470 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:48,470 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:48,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:48,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,473 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,473 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,473 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,473 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:48,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,478 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:48,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:48,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:48,479 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,479 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:48,480 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:48,480 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:48,483 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2113ca0e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:48,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:48,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,488 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:48,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:48,495 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:48,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:48,495 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:48,497 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,497 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,497 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/b21258620abb4e5ba364f3bd4922678c 2023-04-12 13:53:48,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:48,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:48,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:48,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:48,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:48,508 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:48,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:48,516 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/d77cbe5295f6431dab1e637ea8a4d23e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/d77cbe5295f6431dab1e637ea8a4d23e 2023-04-12 13:53:48,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/d77cbe5295f6431dab1e637ea8a4d23e, entries=150, sequenceid=244, filesize=11.8 K 2023-04-12 13:53:48,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/0be9ad9f1db8491cb3ffdaafa6e4a772 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/0be9ad9f1db8491cb3ffdaafa6e4a772 2023-04-12 13:53:48,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/0be9ad9f1db8491cb3ffdaafa6e4a772, entries=150, sequenceid=244, filesize=11.8 K 2023-04-12 13:53:48,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/b21258620abb4e5ba364f3bd4922678c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b21258620abb4e5ba364f3bd4922678c 2023-04-12 13:53:48,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b21258620abb4e5ba364f3bd4922678c, entries=150, sequenceid=244, filesize=11.8 K 2023-04-12 13:53:48,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~64.85 KB/66410, heapSize ~178.38 KB/182656, currentSize=120.76 KB/123660 for 61f538e965579a8c5df754be9b63fe6f in 1032ms, sequenceid=244, compaction requested=true 2023-04-12 13:53:48,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:48,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:48,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:48,582 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:48,582 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:48,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:48,585 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:48,585 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:48,585 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,585 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9f07fa1404f749a08e6dfec25a2d8839, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/37fe4113fde7455f858baa0522b66c99, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/0be9ad9f1db8491cb3ffdaafa6e4a772] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=35.1 K 2023-04-12 13:53:48,586 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9f07fa1404f749a08e6dfec25a2d8839, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681307624333 2023-04-12 13:53:48,587 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 37fe4113fde7455f858baa0522b66c99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681307624692 2023-04-12 13:53:48,587 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 0be9ad9f1db8491cb3ffdaafa6e4a772, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681307626923 2023-04-12 13:53:48,597 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:48,605 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:48,605 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,605 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/caa9582f7aa0431fb7d70e38505684a7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b87b8eab854b440fb32d06e7c0261021, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/d77cbe5295f6431dab1e637ea8a4d23e] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=35.1 K 2023-04-12 13:53:48,607 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting caa9582f7aa0431fb7d70e38505684a7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681307624333 2023-04-12 13:53:48,607 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b87b8eab854b440fb32d06e7c0261021, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681307624692 2023-04-12 13:53:48,608 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:48,609 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d77cbe5295f6431dab1e637ea8a4d23e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681307626923 2023-04-12 13:53:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:48,611 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:48,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:48,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:48,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:48,636 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:48,636 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,637 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:48,637 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:48,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,639 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,639 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:48,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,641 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,641 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:48,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:48,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:48,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:48,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:48,653 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-12 13:53:48,653 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:48,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:48,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:48,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:48,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:48,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:48,662 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:48,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:48,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:48,739 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/4f01b9ebc06d4e6a9e3f5e3cd2331440 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f01b9ebc06d4e6a9e3f5e3cd2331440 2023-04-12 13:53:48,743 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/38dec00d3ae943fbad2232778e1639c7 2023-04-12 13:53:48,759 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into 4f01b9ebc06d4e6a9e3f5e3cd2331440(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:48,759 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:48,759 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307628582; duration=0sec 2023-04-12 13:53:48,759 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:48,759 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:48,762 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:48,762 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:48,762 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,763 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/48ddb73826094bb3bc50ee9c387e0f70, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/28516c23bb3a41dca56a26d87d7a984f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b21258620abb4e5ba364f3bd4922678c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=35.1 K 2023-04-12 13:53:48,766 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 48ddb73826094bb3bc50ee9c387e0f70, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=92, earliestPutTs=1681307624333 2023-04-12 13:53:48,769 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 28516c23bb3a41dca56a26d87d7a984f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681307624692 2023-04-12 13:53:48,770 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b21258620abb4e5ba364f3bd4922678c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681307626923 2023-04-12 13:53:48,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307688761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307688765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307688769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307688770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307688771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,786 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#25 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:48,839 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/c9d4c41faf7f4f3f82251fb8d90741d2 2023-04-12 13:53:48,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307688879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307688879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307688881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:48,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307688885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,895 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/eca514388f2f410aadef1e7f01b055f6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/eca514388f2f410aadef1e7f01b055f6 2023-04-12 13:53:48,921 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into eca514388f2f410aadef1e7f01b055f6(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:48,921 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:48,921 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307628582; duration=0sec 2023-04-12 13:53:48,921 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:48,922 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/ef3998180fa2432ebab452c79807c98c 2023-04-12 13:53:48,931 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/38dec00d3ae943fbad2232778e1639c7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/38dec00d3ae943fbad2232778e1639c7 2023-04-12 13:53:48,940 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/38dec00d3ae943fbad2232778e1639c7, entries=150, sequenceid=278, filesize=11.9 K 2023-04-12 13:53:48,953 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/c9d4c41faf7f4f3f82251fb8d90741d2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/c9d4c41faf7f4f3f82251fb8d90741d2 2023-04-12 13:53:48,966 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/c9d4c41faf7f4f3f82251fb8d90741d2, entries=150, sequenceid=278, filesize=11.9 K 2023-04-12 13:53:48,971 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/ef3998180fa2432ebab452c79807c98c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef3998180fa2432ebab452c79807c98c 2023-04-12 13:53:48,987 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef3998180fa2432ebab452c79807c98c, entries=150, sequenceid=278, filesize=11.9 K 2023-04-12 13:53:48,990 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 61f538e965579a8c5df754be9b63fe6f in 337ms, sequenceid=278, compaction requested=false 2023-04-12 13:53:48,990 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:48,990 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,996 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:48,997 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:48,997 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:48,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45eeb09a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:48,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:49,000 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,002 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:49,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:49,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,003 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:49,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:49,005 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:49,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:49,010 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:49,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:49,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:49,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:49,010 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:49,019 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,019 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:49,019 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:49,020 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:49,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:49,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:49,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:49,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:49,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:49,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:49,027 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:49,027 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:49,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:49,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:49,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:53:49,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:49,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:49,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:49,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:49,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:49,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:49,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/318350f0bca0491486f37ba5abf042b1 2023-04-12 13:53:49,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307689113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,120 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/f56864acd85940dba3932ad030515ca6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f56864acd85940dba3932ad030515ca6 2023-04-12 13:53:49,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307689116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307689118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307689118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,127 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:49,133 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into f56864acd85940dba3932ad030515ca6(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:49,133 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:49,133 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307628582; duration=0sec 2023-04-12 13:53:49,134 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:49,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:49,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:49,153 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:49,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:49,167 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:49,167 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:49,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:49,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:49,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/e1e064fb81144ca5950b3ad008783467 2023-04-12 13:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:49,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:49,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:49,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:49,180 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:49,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,183 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:49,183 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:49,183 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:49,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:49,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bf7cef[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:49,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:49,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:49,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:49,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:49,204 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:49,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:49,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:49,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:49,204 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:49,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:49,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,208 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/82adc42f39364a94a4396a300ddd79a0 2023-04-12 13:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:49,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307689219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307689223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:49,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,226 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:49,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:49,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:49,226 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/318350f0bca0491486f37ba5abf042b1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/318350f0bca0491486f37ba5abf042b1 2023-04-12 13:53:49,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:49,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307689230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:49,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:49,232 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:49,231 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307689231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/318350f0bca0491486f37ba5abf042b1, entries=150, sequenceid=293, filesize=11.9 K 2023-04-12 13:53:49,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/e1e064fb81144ca5950b3ad008783467 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e1e064fb81144ca5950b3ad008783467 2023-04-12 13:53:49,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e1e064fb81144ca5950b3ad008783467, entries=150, sequenceid=293, filesize=11.9 K 2023-04-12 13:53:49,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/82adc42f39364a94a4396a300ddd79a0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/82adc42f39364a94a4396a300ddd79a0 2023-04-12 13:53:49,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/82adc42f39364a94a4396a300ddd79a0, entries=150, sequenceid=293, filesize=11.9 K 2023-04-12 13:53:49,272 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 61f538e965579a8c5df754be9b63fe6f in 185ms, sequenceid=293, compaction requested=true 2023-04-12 13:53:49,273 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:49,273 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:49,273 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:49,273 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:49,273 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:49,273 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:49,280 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:49,280 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:49,280 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,281 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f01b9ebc06d4e6a9e3f5e3cd2331440, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/38dec00d3ae943fbad2232778e1639c7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/318350f0bca0491486f37ba5abf042b1] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=35.8 K 2023-04-12 13:53:49,281 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:49,281 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:49,281 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,281 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f56864acd85940dba3932ad030515ca6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/c9d4c41faf7f4f3f82251fb8d90741d2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e1e064fb81144ca5950b3ad008783467] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=35.8 K 2023-04-12 13:53:49,281 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4f01b9ebc06d4e6a9e3f5e3cd2331440, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681307626923 2023-04-12 13:53:49,282 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f56864acd85940dba3932ad030515ca6, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681307626923 2023-04-12 13:53:49,284 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 38dec00d3ae943fbad2232778e1639c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681307627611 2023-04-12 13:53:49,285 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c9d4c41faf7f4f3f82251fb8d90741d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681307627611 2023-04-12 13:53:49,285 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 318350f0bca0491486f37ba5abf042b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681307628746 2023-04-12 13:53:49,285 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e1e064fb81144ca5950b3ad008783467, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681307628746 2023-04-12 13:53:49,311 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:49,317 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#31 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:49,332 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:49,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:49,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:49,341 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:49,341 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:49,344 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:49,344 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:49,345 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,345 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,347 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:49,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:49,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:49,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:49,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:49,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:49,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:49,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:49,352 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:49,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:49,359 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/a5ce04b7f13044988467b882f78bdae1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a5ce04b7f13044988467b882f78bdae1 2023-04-12 13:53:49,366 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/4f5ebf07048f46a0bba0b0d010c20d8f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f5ebf07048f46a0bba0b0d010c20d8f 2023-04-12 13:53:49,385 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into a5ce04b7f13044988467b882f78bdae1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:49,386 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:49,386 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307629273; duration=0sec 2023-04-12 13:53:49,386 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:49,386 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:49,387 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into 4f5ebf07048f46a0bba0b0d010c20d8f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:49,387 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:49,387 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307629273; duration=0sec 2023-04-12 13:53:49,387 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:49,388 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:49,395 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:49,395 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,395 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/eca514388f2f410aadef1e7f01b055f6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef3998180fa2432ebab452c79807c98c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/82adc42f39364a94a4396a300ddd79a0] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=35.8 K 2023-04-12 13:53:49,396 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting eca514388f2f410aadef1e7f01b055f6, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681307626923 2023-04-12 13:53:49,396 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ef3998180fa2432ebab452c79807c98c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1681307627611 2023-04-12 13:53:49,397 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 82adc42f39364a94a4396a300ddd79a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681307628746 2023-04-12 13:53:49,407 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/2627ae30882c4b46a6bc8f661914abe1 2023-04-12 13:53:49,417 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#33 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:49,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:49,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:49,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307689447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307689448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307689449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307689449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307689553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307689555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307689556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307689557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307689757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307689758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307689758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307689759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:49,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307689781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,847 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/e38f8f83415c42cab187d28daa056184 2023-04-12 13:53:49,860 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/84b1d11dd05246699e54151c6e4ae0fc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/84b1d11dd05246699e54151c6e4ae0fc 2023-04-12 13:53:49,873 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 84b1d11dd05246699e54151c6e4ae0fc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:49,873 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:49,873 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307629273; duration=0sec 2023-04-12 13:53:49,874 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:49,904 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/a702d5978989413aa279ce390ba6800c 2023-04-12 13:53:49,913 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/2627ae30882c4b46a6bc8f661914abe1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2627ae30882c4b46a6bc8f661914abe1 2023-04-12 13:53:49,936 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2627ae30882c4b46a6bc8f661914abe1, entries=150, sequenceid=313, filesize=11.9 K 2023-04-12 13:53:49,938 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/e38f8f83415c42cab187d28daa056184 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e38f8f83415c42cab187d28daa056184 2023-04-12 13:53:49,946 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e38f8f83415c42cab187d28daa056184, entries=150, sequenceid=313, filesize=11.9 K 2023-04-12 13:53:49,946 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-12 13:53:49,948 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/a702d5978989413aa279ce390ba6800c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a702d5978989413aa279ce390ba6800c 2023-04-12 13:53:49,969 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a702d5978989413aa279ce390ba6800c, entries=150, sequenceid=313, filesize=11.9 K 2023-04-12 13:53:49,970 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 61f538e965579a8c5df754be9b63fe6f in 619ms, sequenceid=313, compaction requested=false 2023-04-12 13:53:49,970 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:49,971 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:49,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:49,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:49,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:49,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:49,974 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:49,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8fef7c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:49,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:49,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:49,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:49,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:49,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:49,992 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:49,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:49,995 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:49,995 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:49,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:49,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:49,998 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:49,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,000 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,011 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,011 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,005 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,011 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,011 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:50,062 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:53:50,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:50,063 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:50,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:50,063 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:50,063 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:50,063 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:50,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/7575df5a632d4f3ea318d6be793fac3a 2023-04-12 13:53:50,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307690082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307690097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307690097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307690099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,128 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,129 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,129 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,141 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,141 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,141 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,141 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,142 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,145 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12731379[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,155 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,155 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,156 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,157 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,157 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,159 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:50,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,166 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,166 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,166 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,166 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,166 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307690198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307690203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307690205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307690206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,266 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,271 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,271 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,271 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,271 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,272 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,272 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,273 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,274 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,274 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,274 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,277 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,277 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,278 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,278 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,278 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,280 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a90bcd[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,283 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,283 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,283 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,286 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,288 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,288 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,296 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,296 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,297 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,298 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,298 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,298 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:50,298 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,298 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307690405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,405 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,405 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,405 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,406 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,406 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,406 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,408 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307690407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307690415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307690415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,435 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18cb5373[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,447 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,447 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,447 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,447 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,447 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,453 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,458 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,458 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,459 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,460 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,460 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/21405e8f3ac247a3af0929f1777849b4 2023-04-12 13:53:50,560 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,568 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6dab19[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,575 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,582 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,585 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,585 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,587 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,589 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,593 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,594 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,594 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,694 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,699 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,704 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,704 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,704 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,704 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408f91f6[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,710 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,710 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,711 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,711 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307690709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307690711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,715 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,715 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,715 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,715 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,715 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,719 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307690722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:50,725 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,725 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,726 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:50,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307690723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,826 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,832 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,832 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,832 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,832 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,833 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,833 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,835 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,836 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,836 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,836 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,840 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,842 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a120d49[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,846 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,847 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,852 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,853 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,857 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,859 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,860 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:50,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:50,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:50,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:50,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/4935c4f88c484567903a546b252341f6 2023-04-12 13:53:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:50,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:50,965 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:50,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:50,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:50,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:50,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:50,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:50,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:50,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:50,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:50,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:50,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/7575df5a632d4f3ea318d6be793fac3a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/7575df5a632d4f3ea318d6be793fac3a 2023-04-12 13:53:50,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680093b3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:50,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:50,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,980 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:50,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:50,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:50,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/7575df5a632d4f3ea318d6be793fac3a, entries=200, sequenceid=331, filesize=14.3 K 2023-04-12 13:53:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/21405e8f3ac247a3af0929f1777849b4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21405e8f3ac247a3af0929f1777849b4 2023-04-12 13:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:50,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:50,985 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:50,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:50,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:50,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:50,986 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:50,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:50,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,993 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21405e8f3ac247a3af0929f1777849b4, entries=150, sequenceid=331, filesize=11.9 K 2023-04-12 13:53:50,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/4935c4f88c484567903a546b252341f6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/4935c4f88c484567903a546b252341f6 2023-04-12 13:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:50,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:50,998 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:50,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:50,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:50,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:50,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:50,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:50,999 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:51,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/4935c4f88c484567903a546b252341f6, entries=150, sequenceid=331, filesize=11.9 K 2023-04-12 13:53:51,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 61f538e965579a8c5df754be9b63fe6f in 954ms, sequenceid=331, compaction requested=true 2023-04-12 13:53:51,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:51,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:51,016 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:51,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:51,016 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:51,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:51,019 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:51,019 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:51,019 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:51,019 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:51,019 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a5ce04b7f13044988467b882f78bdae1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e38f8f83415c42cab187d28daa056184, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21405e8f3ac247a3af0929f1777849b4] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.0 K 2023-04-12 13:53:51,019 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:51,020 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:51,020 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f5ebf07048f46a0bba0b0d010c20d8f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2627ae30882c4b46a6bc8f661914abe1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/7575df5a632d4f3ea318d6be793fac3a] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=38.4 K 2023-04-12 13:53:51,020 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a5ce04b7f13044988467b882f78bdae1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681307628746 2023-04-12 13:53:51,021 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4f5ebf07048f46a0bba0b0d010c20d8f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681307628746 2023-04-12 13:53:51,024 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e38f8f83415c42cab187d28daa056184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681307629110 2023-04-12 13:53:51,024 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2627ae30882c4b46a6bc8f661914abe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681307629110 2023-04-12 13:53:51,024 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 21405e8f3ac247a3af0929f1777849b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681307629446 2023-04-12 13:53:51,025 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 7575df5a632d4f3ea318d6be793fac3a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681307629445 2023-04-12 13:53:51,062 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:51,062 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:51,088 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/f04de0d6b9a542d7b1f7023098e87e8d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/f04de0d6b9a542d7b1f7023098e87e8d 2023-04-12 13:53:51,097 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into f04de0d6b9a542d7b1f7023098e87e8d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:51,097 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:51,097 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307631016; duration=0sec 2023-04-12 13:53:51,098 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:51,098 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:51,099 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:51,100 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:51,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:51,100 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:51,100 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:51,100 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/84b1d11dd05246699e54151c6e4ae0fc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a702d5978989413aa279ce390ba6800c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/4935c4f88c484567903a546b252341f6] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.0 K 2023-04-12 13:53:51,101 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 84b1d11dd05246699e54151c6e4ae0fc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1681307628746 2023-04-12 13:53:51,103 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a702d5978989413aa279ce390ba6800c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681307629110 2023-04-12 13:53:51,103 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4935c4f88c484567903a546b252341f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681307629446 2023-04-12 13:53:51,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:51,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:51,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:51,116 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:51,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:51,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:51,117 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:51,117 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:51,117 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:51,118 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:51,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:51,118 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:51,119 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,119 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:51,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:51,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:51,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:51,121 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:51,121 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:51,121 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:53:51,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:51,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:51,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:51,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:51,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:51,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:51,128 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#42 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:51,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:51,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:51,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307691237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307691238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307691239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307691240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307691341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307691345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307691345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307691346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,477 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/122226e6995d4ca18e52c25b4b169bb5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/122226e6995d4ca18e52c25b4b169bb5 2023-04-12 13:53:51,484 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into 122226e6995d4ca18e52c25b4b169bb5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:51,485 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:51,485 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307631016; duration=0sec 2023-04-12 13:53:51,485 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:51,541 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/51878aa47333431da511615f56599dac 2023-04-12 13:53:51,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307691546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307691550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307691551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307691551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,559 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/1449c6bbf5bd47b78856ed7d36283fbb as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1449c6bbf5bd47b78856ed7d36283fbb 2023-04-12 13:53:51,567 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 1449c6bbf5bd47b78856ed7d36283fbb(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:51,567 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:51,568 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307631016; duration=0sec 2023-04-12 13:53:51,568 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:51,570 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/77514818806f4e0680b71992e510b47f 2023-04-12 13:53:51,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307691797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,800 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:53:51,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307691852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307691852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307691853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:51,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307691855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:51,999 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/8b7842d65255494482bcee3a01393720 2023-04-12 13:53:52,012 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/51878aa47333431da511615f56599dac as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/51878aa47333431da511615f56599dac 2023-04-12 13:53:52,021 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/51878aa47333431da511615f56599dac, entries=150, sequenceid=349, filesize=11.9 K 2023-04-12 13:53:52,023 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/77514818806f4e0680b71992e510b47f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/77514818806f4e0680b71992e510b47f 2023-04-12 13:53:52,030 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/77514818806f4e0680b71992e510b47f, entries=150, sequenceid=349, filesize=11.9 K 2023-04-12 13:53:52,032 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/8b7842d65255494482bcee3a01393720 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/8b7842d65255494482bcee3a01393720 2023-04-12 13:53:52,039 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/8b7842d65255494482bcee3a01393720, entries=150, sequenceid=349, filesize=11.9 K 2023-04-12 13:53:52,040 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 61f538e965579a8c5df754be9b63fe6f in 919ms, sequenceid=349, compaction requested=false 2023-04-12 13:53:52,040 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:52,040 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:52,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:52,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f12c8c9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:52,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:52,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,061 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:52,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:52,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:52,072 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:52,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:52,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:52,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:52,072 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:52,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,076 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,078 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,080 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:52,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:52,081 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,081 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:52,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,081 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:52,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,182 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:52,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:52,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:52,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:52,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:52,191 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:52,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:52,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:52,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,193 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:52,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:52,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:52,200 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:52,200 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,200 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:52,200 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:53:52,201 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:52,201 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:52,201 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:52,201 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:52,202 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:52,202 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:52,221 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/8944134af2034a5281896f7c769462af 2023-04-12 13:53:52,257 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/7a95cb4b0e2b4574b80abf0a86af1d0d 2023-04-12 13:53:52,305 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/c493c024dea94b8e9b1f076b8af87de6 2023-04-12 13:53:52,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/8944134af2034a5281896f7c769462af as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8944134af2034a5281896f7c769462af 2023-04-12 13:53:52,325 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8944134af2034a5281896f7c769462af, entries=150, sequenceid=366, filesize=11.9 K 2023-04-12 13:53:52,328 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/7a95cb4b0e2b4574b80abf0a86af1d0d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/7a95cb4b0e2b4574b80abf0a86af1d0d 2023-04-12 13:53:52,342 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/7a95cb4b0e2b4574b80abf0a86af1d0d, entries=150, sequenceid=366, filesize=11.9 K 2023-04-12 13:53:52,343 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/c493c024dea94b8e9b1f076b8af87de6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c493c024dea94b8e9b1f076b8af87de6 2023-04-12 13:53:52,361 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:52,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:52,366 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c493c024dea94b8e9b1f076b8af87de6, entries=150, sequenceid=366, filesize=11.9 K 2023-04-12 13:53:52,367 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=13.42 KB/13740 for 61f538e965579a8c5df754be9b63fe6f in 167ms, sequenceid=366, compaction requested=true 2023-04-12 13:53:52,367 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:52,367 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:52,367 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:52,372 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:52,376 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,377 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:52,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:53:52,388 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:52,388 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:52,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:52,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:52,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732ee0fc[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:52,391 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:52,391 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,393 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:52,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:52,395 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:52,400 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:52,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:52,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:52,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:52,400 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:52,402 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,402 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,402 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,403 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,406 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,410 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:52,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:52,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,411 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:52,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,411 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:52,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307632402, max timestamp=1681307632420Num uniques 859; 2023-04-12 13:53:52,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:53:52,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307632402, max timestamp=1681307632420Num uniques 859; 2023-04-12 13:53:52,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:53:52,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307632402, max timestamp=1681307632420Num uniques 859; 2023-04-12 13:53:52,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:53:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307692451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307692450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307692452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307692452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,512 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:52,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:52,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:52,517 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:52,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:52,517 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:52,518 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,518 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,519 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:52,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:52,521 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,522 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:52,522 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:52,522 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:52,523 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:52,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ecd395[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:52,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:52,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,534 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:52,534 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:52,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:52,534 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:52,537 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:52,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:52,541 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:52,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:52,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:52,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:52,541 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:52,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:52,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:52,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:52,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,555 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:52,555 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:52,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55648 deadline: 1681307692557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307692557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55666 deadline: 1681307692557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55668 deadline: 1681307692557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,655 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:52,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:52,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:52,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:52,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:52,660 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:52,660 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,660 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:52,660 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:52,660 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,662 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,662 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,662 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:52,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:52,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:52,665 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:52,668 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,668 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:52,668 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:52,668 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,671 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,671 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251d5c10[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:52,674 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:52,674 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,677 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:52,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:52,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:52,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:52,684 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:52,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:52,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:52,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:52,684 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:52,687 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,687 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,687 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:52,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,702 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:52,702 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:52,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:52,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307692761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307692761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307692762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307692763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,803 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:52,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:52,808 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:52,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:52,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:52,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,810 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:52,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:52,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:52,813 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,813 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:52,813 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:52,813 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:52,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,815 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c909cfa[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:52,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:52,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:52,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:52,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:52,821 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,821 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:52,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:52,826 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:52,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:52,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:52,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:52,827 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:52,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:52,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/b9dea06c947d4b23a78e0b07fabdec91 2023-04-12 13:53:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:52,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:52,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:52,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:52,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:52,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:52,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:52,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:52,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:52,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:52,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/184357bae3f04fc8baf6271b82541793 2023-04-12 13:53:52,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/cc8abeb6e2264220a8c527879da8aff0 2023-04-12 13:53:52,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/b9dea06c947d4b23a78e0b07fabdec91 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b9dea06c947d4b23a78e0b07fabdec91 2023-04-12 13:53:52,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b9dea06c947d4b23a78e0b07fabdec91, entries=150, sequenceid=377, filesize=11.9 K 2023-04-12 13:53:52,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/184357bae3f04fc8baf6271b82541793 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/184357bae3f04fc8baf6271b82541793 2023-04-12 13:53:52,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/184357bae3f04fc8baf6271b82541793, entries=150, sequenceid=377, filesize=11.9 K 2023-04-12 13:53:52,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/cc8abeb6e2264220a8c527879da8aff0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/cc8abeb6e2264220a8c527879da8aff0 2023-04-12 13:53:52,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/cc8abeb6e2264220a8c527879da8aff0, entries=150, sequenceid=377, filesize=11.9 K 2023-04-12 13:53:52,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 61f538e965579a8c5df754be9b63fe6f in 554ms, sequenceid=377, compaction requested=true 2023-04-12 13:53:52,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:52,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:52,943 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:53:52,943 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:53:52,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:52,943 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:52,944 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:52,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:52,945 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:53:52,945 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:52,946 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,946 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/f04de0d6b9a542d7b1f7023098e87e8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/51878aa47333431da511615f56599dac, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8944134af2034a5281896f7c769462af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b9dea06c947d4b23a78e0b07fabdec91] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=48.0 K 2023-04-12 13:53:52,946 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:53:52,946 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:52,946 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,946 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/122226e6995d4ca18e52c25b4b169bb5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/77514818806f4e0680b71992e510b47f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/7a95cb4b0e2b4574b80abf0a86af1d0d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/184357bae3f04fc8baf6271b82541793] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=48.0 K 2023-04-12 13:53:52,947 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f04de0d6b9a542d7b1f7023098e87e8d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681307629446 2023-04-12 13:53:52,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:52,952 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 122226e6995d4ca18e52c25b4b169bb5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681307629446 2023-04-12 13:53:52,953 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 77514818806f4e0680b71992e510b47f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1681307630075 2023-04-12 13:53:52,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:52,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:52,953 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:52,953 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,953 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:52,953 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:52,953 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7a95cb4b0e2b4574b80abf0a86af1d0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681307631235 2023-04-12 13:53:52,954 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,954 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,954 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 51878aa47333431da511615f56599dac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1681307630075 2023-04-12 13:53:52,955 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 8944134af2034a5281896f7c769462af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681307631235 2023-04-12 13:53:52,955 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b9dea06c947d4b23a78e0b07fabdec91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681307632384 2023-04-12 13:53:52,956 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 184357bae3f04fc8baf6271b82541793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681307632384 2023-04-12 13:53:52,956 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:52,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:52,956 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,958 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:52,958 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:52,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:52,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:52,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:52,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:52,988 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:52,997 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:52,997 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:52,997 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:52,998 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:52,998 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 13:53:52,998 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:52,998 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:53:53,038 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/4e666144cee64e619593cd5a72703b94 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4e666144cee64e619593cd5a72703b94 2023-04-12 13:53:53,048 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into 4e666144cee64e619593cd5a72703b94(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:53,048 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:53,048 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=12, startTime=1681307632943; duration=0sec 2023-04-12 13:53:53,048 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:53,048 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:53:53,052 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:53:53,052 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:53,052 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:53,052 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1449c6bbf5bd47b78856ed7d36283fbb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/8b7842d65255494482bcee3a01393720, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c493c024dea94b8e9b1f076b8af87de6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/cc8abeb6e2264220a8c527879da8aff0] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=48.0 K 2023-04-12 13:53:53,053 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 1449c6bbf5bd47b78856ed7d36283fbb, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681307629446 2023-04-12 13:53:53,055 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 8b7842d65255494482bcee3a01393720, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1681307630075 2023-04-12 13:53:53,056 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c493c024dea94b8e9b1f076b8af87de6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681307631235 2023-04-12 13:53:53,057 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting cc8abeb6e2264220a8c527879da8aff0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681307632384 2023-04-12 13:53:53,065 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/4713e536cc1d4829a7847a9d65e9c718 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4713e536cc1d4829a7847a9d65e9c718 2023-04-12 13:53:53,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:53,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:53,074 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:53,076 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into 4713e536cc1d4829a7847a9d65e9c718(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:53,076 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:53,076 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=12, startTime=1681307632943; duration=0sec 2023-04-12 13:53:53,076 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:53,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307693075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307693076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307693077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307693079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,104 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/85770e8423754fed8aeff129f8ff0c4e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/85770e8423754fed8aeff129f8ff0c4e 2023-04-12 13:53:53,116 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 85770e8423754fed8aeff129f8ff0c4e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:53,116 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:53,116 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=12, startTime=1681307632943; duration=0sec 2023-04-12 13:53:53,116 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:53,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307693180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307693183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307693183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307693186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307693384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307693386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307693387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307693388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,445 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/70196cf78a42496798c676c388811efd 2023-04-12 13:53:53,490 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/a219bf818e05481f841e49fdac632e9f 2023-04-12 13:53:53,523 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/46c5ab36e9644a6581f36de1cd897de1 2023-04-12 13:53:53,530 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/70196cf78a42496798c676c388811efd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/70196cf78a42496798c676c388811efd 2023-04-12 13:53:53,537 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/70196cf78a42496798c676c388811efd, entries=150, sequenceid=412, filesize=11.9 K 2023-04-12 13:53:53,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/a219bf818e05481f841e49fdac632e9f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a219bf818e05481f841e49fdac632e9f 2023-04-12 13:53:53,555 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a219bf818e05481f841e49fdac632e9f, entries=150, sequenceid=412, filesize=11.9 K 2023-04-12 13:53:53,557 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/46c5ab36e9644a6581f36de1cd897de1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/46c5ab36e9644a6581f36de1cd897de1 2023-04-12 13:53:53,564 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/46c5ab36e9644a6581f36de1cd897de1, entries=150, sequenceid=412, filesize=11.9 K 2023-04-12 13:53:53,565 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 61f538e965579a8c5df754be9b63fe6f in 567ms, sequenceid=412, compaction requested=false 2023-04-12 13:53:53,566 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:53,566 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:53,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:53,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:53,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78bdba94[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:53,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:53,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:53,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:53,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:53,580 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:53,580 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:53,586 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:53,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:53,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:53,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:53,586 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:53,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,591 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,592 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,595 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:53,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:53,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,596 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:53,597 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:53,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:53,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:53,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:53:53,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:53,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:53,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:53,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:53,696 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:53,696 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:53,697 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:53,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:53,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:53,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:53,703 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:53,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:53,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:53,705 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:53,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:53,733 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:53,733 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:53,733 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:53,733 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:53,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:53,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/c0bd3cca2f6d4a53bb869d433f820d2e 2023-04-12 13:53:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307693733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5762be6d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:53,739 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:53,739 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,741 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:53,741 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:53,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,742 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,742 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307693734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307693735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307693736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,747 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:53,751 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:53,751 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:53,751 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:53,751 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:53,751 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:53,753 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,753 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:53,753 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:53,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:53,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:53,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:53,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:53,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:53,762 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:53,762 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:53,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307693839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307693843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307693843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:53,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307693844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,862 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:53,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:53,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:53,867 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:53,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:53,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:53,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:53,868 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,868 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,869 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:53,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:53,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:53,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:53,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:53,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:53,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:53,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:53,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:53,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:53,880 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,880 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,881 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,881 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d0854d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:53,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:53,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:53,884 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,884 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:53,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:53,886 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:53,889 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:53,889 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:53,889 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:53,889 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:53,889 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:53,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:53,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:53,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,893 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:53,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:53,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:53,900 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:53,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:53,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:53,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:53,901 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:53,901 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:54,001 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:54,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:54,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:54,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:54,005 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:54,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:54,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:54,006 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,006 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,007 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:54,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:54,024 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:54,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:54,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:54,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:54,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605fade1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:54,030 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:54,031 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,032 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:54,032 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:54,032 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,032 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:54,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,034 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:54,037 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:54,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:54,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:54,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:54,037 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:54,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,047 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307694046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307694048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307694048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307694049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:54,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,060 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:54,060 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,160 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:54,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:54,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:54,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:54,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:54,165 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:54,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,166 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:54,166 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:54,166 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,166 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:54,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:54,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:54,170 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,170 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:54,170 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:54,170 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:54,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:54,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:54,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:54,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a3d264[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:54,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:54,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:54,176 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:54,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/01f3efceeda742eca1a2720f4ee57fae 2023-04-12 13:53:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:54,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:54,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:54,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,194 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,205 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:54,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:54,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:54,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:54,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:54,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:54,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:54,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:54,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:54,310 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:54,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:54,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:54,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,313 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,313 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,313 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:54,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:54,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:54,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:54,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:54,315 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,315 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:54,315 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:54,315 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:54,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:54,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,317 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:54,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f08b6b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:54,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:54,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,319 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:54,319 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:54,319 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,319 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:54,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:54,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:54,323 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:54,323 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:54,323 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:54,323 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:54,323 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:54,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,327 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,329 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:54,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,331 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,331 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,332 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307694350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307694353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307694355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307694359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,432 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:54,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:54,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:54,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:54,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:54,438 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:54,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:54,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:54,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:54,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:54,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:54,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:54,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:54,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:54,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:54,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,461 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:54,461 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:54,462 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,463 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,463 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:54,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b169a35[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:54,464 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:54,465 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,472 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:54,472 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:54,475 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,475 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:54,482 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:54,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:54,486 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:54,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:54,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:54,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:54,486 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:54,488 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,488 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,488 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,488 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,491 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:54,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:54,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,495 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:54,495 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,495 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,595 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:54,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:54,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:54,604 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:54,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:54,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:54,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,607 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,608 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,610 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:54,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:54,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:54,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,611 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:54,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:54,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:54,614 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:54,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bb0a13[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:54,616 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:54,617 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,618 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:54,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:54,619 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,619 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,620 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:54,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/80352f072f62454fa0a82467313dc4f2 2023-04-12 13:53:54,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/c0bd3cca2f6d4a53bb869d433f820d2e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0bd3cca2f6d4a53bb869d433f820d2e 2023-04-12 13:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:54,640 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:54,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:54,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:54,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:54,640 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:54,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:54,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0bd3cca2f6d4a53bb869d433f820d2e, entries=150, sequenceid=429, filesize=11.9 K 2023-04-12 13:53:54,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/01f3efceeda742eca1a2720f4ee57fae as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/01f3efceeda742eca1a2720f4ee57fae 2023-04-12 13:53:54,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:54,655 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/01f3efceeda742eca1a2720f4ee57fae, entries=150, sequenceid=429, filesize=11.9 K 2023-04-12 13:53:54,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/80352f072f62454fa0a82467313dc4f2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/80352f072f62454fa0a82467313dc4f2 2023-04-12 13:53:54,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:54,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:54,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:54,661 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,662 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,662 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:54,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:54,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/80352f072f62454fa0a82467313dc4f2, entries=150, sequenceid=429, filesize=11.9 K 2023-04-12 13:53:54,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 61f538e965579a8c5df754be9b63fe6f in 982ms, sequenceid=429, compaction requested=true 2023-04-12 13:53:54,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:54,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:54,677 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:54,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:54,677 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:54,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:54,679 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:54,679 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:54,679 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,679 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4e666144cee64e619593cd5a72703b94, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/70196cf78a42496798c676c388811efd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0bd3cca2f6d4a53bb869d433f820d2e] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.3 K 2023-04-12 13:53:54,680 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:54,680 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:54,680 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,680 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4713e536cc1d4829a7847a9d65e9c718, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a219bf818e05481f841e49fdac632e9f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/01f3efceeda742eca1a2720f4ee57fae] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.3 K 2023-04-12 13:53:54,681 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4e666144cee64e619593cd5a72703b94, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681307632384 2023-04-12 13:53:54,681 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4713e536cc1d4829a7847a9d65e9c718, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681307632384 2023-04-12 13:53:54,681 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 70196cf78a42496798c676c388811efd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681307632448 2023-04-12 13:53:54,681 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a219bf818e05481f841e49fdac632e9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681307632448 2023-04-12 13:53:54,682 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c0bd3cca2f6d4a53bb869d433f820d2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681307633072 2023-04-12 13:53:54,682 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 01f3efceeda742eca1a2720f4ee57fae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681307633072 2023-04-12 13:53:54,693 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:54,693 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#61 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:54,762 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:54,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:54,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:54,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:54,766 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:54,766 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,766 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:54,766 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:54,766 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,767 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,768 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:54,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:54,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:54,776 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,776 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:54,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:54,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:54,777 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:54,777 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:54,778 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:54,778 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:54,778 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:54,778 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:54,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:54,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307694875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307694876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307694877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307694884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307694981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307694981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307694985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:54,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:54,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307694988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,136 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/ab0f8524f7c54d4597969d8f14c7f62f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ab0f8524f7c54d4597969d8f14c7f62f 2023-04-12 13:53:55,144 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/fe758fe19c3f4c5989fda2b8ab65177c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/fe758fe19c3f4c5989fda2b8ab65177c 2023-04-12 13:53:55,146 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into ab0f8524f7c54d4597969d8f14c7f62f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:55,146 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:55,146 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307634677; duration=0sec 2023-04-12 13:53:55,147 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:55,147 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:55,149 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:55,149 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:55,149 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,149 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/85770e8423754fed8aeff129f8ff0c4e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/46c5ab36e9644a6581f36de1cd897de1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/80352f072f62454fa0a82467313dc4f2] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.3 K 2023-04-12 13:53:55,150 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 85770e8423754fed8aeff129f8ff0c4e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1681307632384 2023-04-12 13:53:55,151 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 46c5ab36e9644a6581f36de1cd897de1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1681307632448 2023-04-12 13:53:55,151 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 80352f072f62454fa0a82467313dc4f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681307633072 2023-04-12 13:53:55,154 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into fe758fe19c3f4c5989fda2b8ab65177c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:55,154 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:55,154 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307634677; duration=0sec 2023-04-12 13:53:55,154 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:55,162 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:55,182 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/7b6926c290994bda8ce10507842b1766 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7b6926c290994bda8ce10507842b1766 2023-04-12 13:53:55,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307695184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307695187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307695189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307695191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,195 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 7b6926c290994bda8ce10507842b1766(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:55,195 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:55,195 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307634677; duration=0sec 2023-04-12 13:53:55,196 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:55,204 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/bc6c0c7bd9b745cf9d0f6510c1afcc93 2023-04-12 13:53:55,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307695489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307695489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307695492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307695493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,627 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/248f70215109450a881d8aee549802a6 2023-04-12 13:53:55,652 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/6ab4e95718674f0fbdf7cdf556d406cd 2023-04-12 13:53:55,666 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/bc6c0c7bd9b745cf9d0f6510c1afcc93 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/bc6c0c7bd9b745cf9d0f6510c1afcc93 2023-04-12 13:53:55,675 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/bc6c0c7bd9b745cf9d0f6510c1afcc93, entries=150, sequenceid=447, filesize=11.9 K 2023-04-12 13:53:55,676 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/248f70215109450a881d8aee549802a6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/248f70215109450a881d8aee549802a6 2023-04-12 13:53:55,686 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/248f70215109450a881d8aee549802a6, entries=150, sequenceid=447, filesize=11.9 K 2023-04-12 13:53:55,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/6ab4e95718674f0fbdf7cdf556d406cd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6ab4e95718674f0fbdf7cdf556d406cd 2023-04-12 13:53:55,694 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6ab4e95718674f0fbdf7cdf556d406cd, entries=150, sequenceid=447, filesize=11.9 K 2023-04-12 13:53:55,695 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 61f538e965579a8c5df754be9b63fe6f in 918ms, sequenceid=447, compaction requested=false 2023-04-12 13:53:55,695 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:55,695 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:55,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:55,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d60de52[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:55,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:55,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:55,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:55,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:55,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:55,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:55,709 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:55,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:55,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:55,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:55,709 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:55,710 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,711 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:55,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,711 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:55,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:55,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:55,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,717 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,717 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,717 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:55,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:55,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:55,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:55,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:55,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:55,718 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:55,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:55,718 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:55,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:55,818 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:55,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:55,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:55,826 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:55,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:55,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:55,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:55,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-12 13:53:55,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:55,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:55,827 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:55,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:55,827 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:55,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:55,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:55,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:55,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:55,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:55,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:55,831 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,831 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:55,832 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:55,832 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:55,833 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:55,834 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,834 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:55,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:55,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e78ad46[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:55,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:55,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,839 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:55,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:55,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:55,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:55,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:55,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:55,845 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:55,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:55,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:55,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:55,846 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:55,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:55,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:55,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/9a66cf714d914330ba87a38ea4879c36 2023-04-12 13:53:55,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:55,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:55,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:55,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:55,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:55,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:55,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:55,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:55,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,861 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:55,861 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:55,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:55,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:55,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307695891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/d165fa14bbc049febfcaa31a0b54fcd9 2023-04-12 13:53:55,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/372de978db6145f9953b38049adffeb2 2023-04-12 13:53:55,922 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/9a66cf714d914330ba87a38ea4879c36 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/9a66cf714d914330ba87a38ea4879c36 2023-04-12 13:53:55,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/9a66cf714d914330ba87a38ea4879c36, entries=200, sequenceid=466, filesize=14.3 K 2023-04-12 13:53:55,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/d165fa14bbc049febfcaa31a0b54fcd9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d165fa14bbc049febfcaa31a0b54fcd9 2023-04-12 13:53:55,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d165fa14bbc049febfcaa31a0b54fcd9, entries=150, sequenceid=466, filesize=11.9 K 2023-04-12 13:53:55,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/372de978db6145f9953b38049adffeb2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/372de978db6145f9953b38049adffeb2 2023-04-12 13:53:55,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/372de978db6145f9953b38049adffeb2, entries=150, sequenceid=466, filesize=11.9 K 2023-04-12 13:53:55,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 61f538e965579a8c5df754be9b63fe6f in 125ms, sequenceid=466, compaction requested=true 2023-04-12 13:53:55,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:55,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:55,952 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:55,952 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:55,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:55,952 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:55,953 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:55,953 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:55,953 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:55,953 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:55,953 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,953 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,953 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ab0f8524f7c54d4597969d8f14c7f62f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/bc6c0c7bd9b745cf9d0f6510c1afcc93, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/9a66cf714d914330ba87a38ea4879c36] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=38.8 K 2023-04-12 13:53:55,953 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/fe758fe19c3f4c5989fda2b8ab65177c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/248f70215109450a881d8aee549802a6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d165fa14bbc049febfcaa31a0b54fcd9] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.4 K 2023-04-12 13:53:55,954 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ab0f8524f7c54d4597969d8f14c7f62f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681307633072 2023-04-12 13:53:55,954 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting fe758fe19c3f4c5989fda2b8ab65177c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681307633072 2023-04-12 13:53:55,954 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bc6c0c7bd9b745cf9d0f6510c1afcc93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1681307633709 2023-04-12 13:53:55,954 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 248f70215109450a881d8aee549802a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1681307633709 2023-04-12 13:53:55,955 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 9a66cf714d914330ba87a38ea4879c36, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681307634873 2023-04-12 13:53:55,956 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d165fa14bbc049febfcaa31a0b54fcd9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681307634873 2023-04-12 13:53:55,961 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:55,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:55,965 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:55,965 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#70 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:55,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:55,969 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:55,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:55,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:55,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:55,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:55,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:55,975 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:55,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:55,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:55,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:55,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:55,977 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:55,984 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-12 13:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:55,985 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:55,985 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:55,985 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:55,985 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:55,985 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:55,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:55,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:56,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307696009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307696009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307696009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307696009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307696010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,015 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/a77dacd9b5034dc4b8c2e10c0d776c44 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a77dacd9b5034dc4b8c2e10c0d776c44 2023-04-12 13:53:56,018 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/efdf2644c2c7494489036e5fb08dd0b8 2023-04-12 13:53:56,022 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into a77dacd9b5034dc4b8c2e10c0d776c44(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:56,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:56,022 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307635952; duration=0sec 2023-04-12 13:53:56,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:56,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:56,024 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:56,024 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:56,024 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,025 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7b6926c290994bda8ce10507842b1766, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6ab4e95718674f0fbdf7cdf556d406cd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/372de978db6145f9953b38049adffeb2] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.4 K 2023-04-12 13:53:56,025 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 7b6926c290994bda8ce10507842b1766, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681307633072 2023-04-12 13:53:56,026 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6ab4e95718674f0fbdf7cdf556d406cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1681307633709 2023-04-12 13:53:56,026 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 372de978db6145f9953b38049adffeb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681307634873 2023-04-12 13:53:56,041 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:56,051 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/79d72f139e654fd1afdaee9a6bea2832 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/79d72f139e654fd1afdaee9a6bea2832 2023-04-12 13:53:56,057 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 79d72f139e654fd1afdaee9a6bea2832(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:56,057 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:56,057 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307635952; duration=0sec 2023-04-12 13:53:56,057 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:56,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307696113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307696113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307696114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307696114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307696114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307696315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307696316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307696316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307696317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307696318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,398 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/682a14ca91114402852e3f872ac8e105 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/682a14ca91114402852e3f872ac8e105 2023-04-12 13:53:56,404 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into 682a14ca91114402852e3f872ac8e105(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:56,404 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:56,404 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307635952; duration=0sec 2023-04-12 13:53:56,404 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:56,435 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/97793627139b427390aef7b44e8b88b8 2023-04-12 13:53:56,454 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/74334584c19c4a7e8fcc948550b36d3d 2023-04-12 13:53:56,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/efdf2644c2c7494489036e5fb08dd0b8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/efdf2644c2c7494489036e5fb08dd0b8 2023-04-12 13:53:56,466 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/efdf2644c2c7494489036e5fb08dd0b8, entries=150, sequenceid=482, filesize=11.9 K 2023-04-12 13:53:56,467 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/97793627139b427390aef7b44e8b88b8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/97793627139b427390aef7b44e8b88b8 2023-04-12 13:53:56,473 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/97793627139b427390aef7b44e8b88b8, entries=150, sequenceid=482, filesize=11.9 K 2023-04-12 13:53:56,493 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/74334584c19c4a7e8fcc948550b36d3d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/74334584c19c4a7e8fcc948550b36d3d 2023-04-12 13:53:56,500 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/74334584c19c4a7e8fcc948550b36d3d, entries=150, sequenceid=482, filesize=11.9 K 2023-04-12 13:53:56,503 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 61f538e965579a8c5df754be9b63fe6f in 519ms, sequenceid=482, compaction requested=false 2023-04-12 13:53:56,504 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:56,504 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:56,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:56,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:56,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:56,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:56,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7120458e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:56,510 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:56,510 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,512 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:56,513 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:56,513 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:56,513 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,513 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:56,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,515 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:56,519 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:56,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:56,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:56,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:56,519 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:56,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:56,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,573 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:56,573 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:56,574 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:56,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:56,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:56,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:56,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307696643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307696644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307696653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307696653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307696653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/598fbb62ca0f48f28e9f4826b9281bcd 2023-04-12 13:53:56,674 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:56,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:56,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:56,681 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:56,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:56,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:56,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:56,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:56,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:56,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:56,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:56,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:56,718 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,719 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:56,719 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:56,719 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:56,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:56,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:56,721 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:56,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa5322b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:56,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:56,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,726 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:56,726 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:56,726 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:56,726 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,726 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,728 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:56,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:56,731 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:56,731 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:56,731 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:56,731 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:56,731 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:56,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,735 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,736 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:56,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:56,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:56,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:56,739 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,740 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:56,740 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,740 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:56,740 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:56,740 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307696758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307696758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307696758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307696759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307696759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,840 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:56,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:56,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:56,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:56,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:56,843 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:56,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:56,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:56,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:56,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:56,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:56,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:56,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:56,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:56,847 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,847 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:56,847 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:56,847 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:56,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:56,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446cedc3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:56,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:56,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,855 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:56,855 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:56,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,855 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:56,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:56,859 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:56,859 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:56,859 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:56,859 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:56,859 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:56,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,862 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,863 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:56,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:56,865 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:56,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:56,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,866 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:56,866 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:56,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307696963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307696963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307696964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,967 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:56,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307696965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:56,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307696966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:56,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:56,972 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:56,972 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:56,972 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:56,972 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:56,974 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,974 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:56,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:56,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:56,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:56,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:56,983 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,983 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:56,983 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:56,983 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:56,985 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f556a95[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:56,987 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:56,987 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,988 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:56,989 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,989 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:56,990 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:56,995 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:56,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:56,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:56,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:56,995 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:56,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:56,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:56,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:57,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,005 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:57,005 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,105 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:57,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:57,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:57,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:57,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:57,110 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:57,110 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,110 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:57,110 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:57,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,112 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,112 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:57,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:57,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:57,115 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,115 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:57,115 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:57,115 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,115 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:57,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,121 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,121 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,123 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/4076a3ffaf434990859e58ce7546f831 2023-04-12 13:53:57,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2a5ec4[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:57,125 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:57,125 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,132 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,132 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:57,132 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:57,132 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:57,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:57,134 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:57,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:57,140 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:57,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:57,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:57,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:57,140 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:57,142 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,142 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,142 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:57,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:57,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:57,152 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:57,152 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:57,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,252 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:57,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:57,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:57,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:57,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:57,256 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:57,256 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,256 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:57,256 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:57,256 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,257 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,259 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:57,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:57,261 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,261 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:57,261 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:57,261 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:57,261 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a280300[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:57,265 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:57,265 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,266 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:57,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:57,266 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:57,268 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307697268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:57,271 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:57,271 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:57,271 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:57,271 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:57,271 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:57,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307697269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307697269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307697269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307697269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,274 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,274 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,274 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,275 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,276 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,278 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:57,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:57,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:57,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,279 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,279 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:57,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,279 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:57,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:57,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:57,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:57,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:57,385 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:57,385 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,385 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:57,385 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:57,386 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,386 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,387 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,387 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,387 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:57,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:57,389 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,389 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:57,389 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:57,389 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:57,391 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,391 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,392 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,392 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:57,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@909f173[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:57,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:57,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,397 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:57,397 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:57,399 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:57,399 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,399 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:57,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:57,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:57,412 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:57,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:57,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:57,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:57,412 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:57,414 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,414 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,414 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:57,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:57,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,430 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,431 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:57,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,531 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:57,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:57,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:57,536 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:57,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:57,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:57,540 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,540 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/13e92edad4a646eaa14b6e0018b49727 2023-04-12 13:53:57,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:57,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:57,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:57,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:57,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:57,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/598fbb62ca0f48f28e9f4826b9281bcd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/598fbb62ca0f48f28e9f4826b9281bcd 2023-04-12 13:53:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:57,547 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,547 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:57,547 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:57,547 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:57,552 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/598fbb62ca0f48f28e9f4826b9281bcd, entries=200, sequenceid=502, filesize=14.3 K 2023-04-12 13:53:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/4076a3ffaf434990859e58ce7546f831 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4076a3ffaf434990859e58ce7546f831 2023-04-12 13:53:57,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:57,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcc3305[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:57,555 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:57,555 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,556 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:57,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:57,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,563 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:57,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:57,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:57,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:57,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:57,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:57,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:57,568 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,568 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:57,568 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4076a3ffaf434990859e58ce7546f831, entries=150, sequenceid=502, filesize=11.9 K 2023-04-12 13:53:57,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,570 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/13e92edad4a646eaa14b6e0018b49727 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/13e92edad4a646eaa14b6e0018b49727 2023-04-12 13:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,571 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:57,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,575 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:57,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:57,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:57,575 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:57,576 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:57,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:57,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/13e92edad4a646eaa14b6e0018b49727, entries=150, sequenceid=502, filesize=11.9 K 2023-04-12 13:53:57,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 61f538e965579a8c5df754be9b63fe6f in 958ms, sequenceid=502, compaction requested=true 2023-04-12 13:53:57,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:57,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:57,580 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:57,580 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:57,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:57,581 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:57,582 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:57,582 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39782 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:57,582 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:53:57,582 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:53:57,582 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,582 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,583 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/682a14ca91114402852e3f872ac8e105, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/97793627139b427390aef7b44e8b88b8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4076a3ffaf434990859e58ce7546f831] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.5 K 2023-04-12 13:53:57,583 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a77dacd9b5034dc4b8c2e10c0d776c44, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/efdf2644c2c7494489036e5fb08dd0b8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/598fbb62ca0f48f28e9f4826b9281bcd] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=38.8 K 2023-04-12 13:53:57,583 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 682a14ca91114402852e3f872ac8e105, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681307634873 2023-04-12 13:53:57,583 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a77dacd9b5034dc4b8c2e10c0d776c44, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681307634873 2023-04-12 13:53:57,584 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 97793627139b427390aef7b44e8b88b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681307635872 2023-04-12 13:53:57,584 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting efdf2644c2c7494489036e5fb08dd0b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681307635872 2023-04-12 13:53:57,584 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 598fbb62ca0f48f28e9f4826b9281bcd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307635997 2023-04-12 13:53:57,585 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4076a3ffaf434990859e58ce7546f831, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307636001 2023-04-12 13:53:57,593 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#78 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:57,595 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:57,676 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:57,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:57,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:57,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:57,680 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:57,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:57,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:57,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,682 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:57,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:57,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:57,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:57,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:57,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:57,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:57,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:57,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:57,688 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:57,688 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:57,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:57,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307697790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307697794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307697794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307697795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307697795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307697896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307697901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307697903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307697903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:57,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:57,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307697903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,014 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/173470ed3d7f4275a241341bc1b4a059 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/173470ed3d7f4275a241341bc1b4a059 2023-04-12 13:53:58,022 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into 173470ed3d7f4275a241341bc1b4a059(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:58,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:58,022 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307637580; duration=0sec 2023-04-12 13:53:58,023 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:53:58,022 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/5eab54ed20504fe8980345385d687a54 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5eab54ed20504fe8980345385d687a54 2023-04-12 13:53:58,023 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:53:58,026 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:53:58,026 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:53:58,027 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:58,027 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/79d72f139e654fd1afdaee9a6bea2832, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/74334584c19c4a7e8fcc948550b36d3d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/13e92edad4a646eaa14b6e0018b49727] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.5 K 2023-04-12 13:53:58,028 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 79d72f139e654fd1afdaee9a6bea2832, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681307634873 2023-04-12 13:53:58,028 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 74334584c19c4a7e8fcc948550b36d3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1681307635872 2023-04-12 13:53:58,029 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 13e92edad4a646eaa14b6e0018b49727, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307636001 2023-04-12 13:53:58,033 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into 5eab54ed20504fe8980345385d687a54(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:58,033 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:58,033 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307637580; duration=0sec 2023-04-12 13:53:58,033 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:58,039 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:53:58,098 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/c0c56140ce5d425a9e9fbaa982ff645c 2023-04-12 13:53:58,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307698102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307698104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307698105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307698105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307698106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,113 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/9e0ce3b29a1248629b44e1cf686bfb3a 2023-04-12 13:53:58,124 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/27dd61e0fe0240848321f40a3c08c384 2023-04-12 13:53:58,130 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/c0c56140ce5d425a9e9fbaa982ff645c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0c56140ce5d425a9e9fbaa982ff645c 2023-04-12 13:53:58,135 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0c56140ce5d425a9e9fbaa982ff645c, entries=150, sequenceid=517, filesize=11.9 K 2023-04-12 13:53:58,136 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/9e0ce3b29a1248629b44e1cf686bfb3a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9e0ce3b29a1248629b44e1cf686bfb3a 2023-04-12 13:53:58,141 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9e0ce3b29a1248629b44e1cf686bfb3a, entries=150, sequenceid=517, filesize=11.9 K 2023-04-12 13:53:58,142 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/27dd61e0fe0240848321f40a3c08c384 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/27dd61e0fe0240848321f40a3c08c384 2023-04-12 13:53:58,150 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/27dd61e0fe0240848321f40a3c08c384, entries=150, sequenceid=517, filesize=11.9 K 2023-04-12 13:53:58,152 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 61f538e965579a8c5df754be9b63fe6f in 464ms, sequenceid=517, compaction requested=false 2023-04-12 13:53:58,153 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:58,153 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:58,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:58,156 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:58,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:58,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e8806[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:58,159 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:58,159 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,160 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,161 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,161 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:58,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:58,161 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:58,163 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:58,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:58,173 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:58,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:58,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:58,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:58,174 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:58,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:58,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:58,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:58,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:58,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:58,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:58,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:58,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:58,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:58,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:58,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:58,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:58,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:58,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:58,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:58,286 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:58,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:58,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:58,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:58,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:58,288 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:58,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:58,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:58,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:58,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:58,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:58,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:58,289 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:58,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:58,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:58,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:58,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:58,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:58,291 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:53:58,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:58,292 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:58,292 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:58,292 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:58,292 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:58,292 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:58,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:53:58,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:53:58,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307698417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307698420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307698421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307698422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307698422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,456 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/058649ccd27f44eaa53e76d3d4570a7f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/058649ccd27f44eaa53e76d3d4570a7f 2023-04-12 13:53:58,478 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 058649ccd27f44eaa53e76d3d4570a7f(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:53:58,478 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:58,478 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307637581; duration=0sec 2023-04-12 13:53:58,478 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:53:58,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307698523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307698524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307698525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307698532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55644 deadline: 1681307698532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,702 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/93d1c7fb7cce4c1d8aeb25f8d15667ab 2023-04-12 13:53:58,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307698728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307698729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307698729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307698736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:58,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307698737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307699032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307699035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307699036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307699040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307699042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,123 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/3f49f562150b4d92a06ebcc555ea4f8c 2023-04-12 13:53:59,536 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/b62fd472846b46f4942fbf22a5a88157 2023-04-12 13:53:59,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307699535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307699541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307699541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,544 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/93d1c7fb7cce4c1d8aeb25f8d15667ab as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/93d1c7fb7cce4c1d8aeb25f8d15667ab 2023-04-12 13:53:59,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307699546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:53:59,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307699547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,552 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/93d1c7fb7cce4c1d8aeb25f8d15667ab, entries=150, sequenceid=537, filesize=11.9 K 2023-04-12 13:53:59,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/3f49f562150b4d92a06ebcc555ea4f8c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/3f49f562150b4d92a06ebcc555ea4f8c 2023-04-12 13:53:59,559 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/3f49f562150b4d92a06ebcc555ea4f8c, entries=150, sequenceid=537, filesize=11.9 K 2023-04-12 13:53:59,560 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/b62fd472846b46f4942fbf22a5a88157 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b62fd472846b46f4942fbf22a5a88157 2023-04-12 13:53:59,577 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b62fd472846b46f4942fbf22a5a88157, entries=150, sequenceid=537, filesize=11.9 K 2023-04-12 13:53:59,578 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 61f538e965579a8c5df754be9b63fe6f in 1287ms, sequenceid=537, compaction requested=true 2023-04-12 13:53:59,579 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:53:59,579 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:53:59,579 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:53:59,581 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:59,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:59,583 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,583 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:53:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:59,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:59,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:53:59,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:53:59,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6904ab6e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:53:59,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:53:59,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:53:59,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:53:59,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:53:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:53:59,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:59,593 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:53:59,598 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:53:59,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:53:59,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:53:59,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:53:59,598 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:53:59,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:53:59,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:59,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:53:59,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:59,601 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:53:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:59,603 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:59,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:53:59,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:53:59,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:53:59,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:53:59,606 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:53:59,607 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:53:59,607 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:53:59,707 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:53:59,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:53:59,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:53:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:53:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:53:59,711 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:53:59,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:59,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:53:59,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:53:59,712 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,712 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:59,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:53:59,713 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:53:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:53:59,713 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:53:59,713 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:53:59,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:59,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:53:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:53:59,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:53:59,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:53:59,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:53:59,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:53:59,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:53:59,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:53:59,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:53:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:53:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:53:59,715 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:53:59,715 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:53:59,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:00,124 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/0ce68d4dba85493e8a62592e9e1c36f9 2023-04-12 13:54:00,537 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/18982788da5d44b590faa014b9fd9e89 2023-04-12 13:54:00,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:00,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:00,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307700563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307700565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307700566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307700570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307700570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,590 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/a694cc9466a14c3ebf359d0d30db4f3d 2023-04-12 13:54:00,597 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/0ce68d4dba85493e8a62592e9e1c36f9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/0ce68d4dba85493e8a62592e9e1c36f9 2023-04-12 13:54:00,605 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/0ce68d4dba85493e8a62592e9e1c36f9, entries=150, sequenceid=552, filesize=11.9 K 2023-04-12 13:54:00,607 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/18982788da5d44b590faa014b9fd9e89 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/18982788da5d44b590faa014b9fd9e89 2023-04-12 13:54:00,619 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/18982788da5d44b590faa014b9fd9e89, entries=150, sequenceid=552, filesize=11.9 K 2023-04-12 13:54:00,624 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/a694cc9466a14c3ebf359d0d30db4f3d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a694cc9466a14c3ebf359d0d30db4f3d 2023-04-12 13:54:00,632 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a694cc9466a14c3ebf359d0d30db4f3d, entries=150, sequenceid=552, filesize=11.9 K 2023-04-12 13:54:00,633 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 61f538e965579a8c5df754be9b63fe6f in 918ms, sequenceid=552, compaction requested=true 2023-04-12 13:54:00,633 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:00,633 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:00,633 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:00,636 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,636 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:00,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:00,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc05d54[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:00,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,640 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:00,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:00,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:00,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:00,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:00,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:00,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:00,643 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:00,646 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:00,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:00,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:00,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:00,647 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:00,648 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,648 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,648 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:00,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,657 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:00,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:00,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:00,673 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:54:00,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:54:00,673 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:00,673 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:54:00,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:00,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:54:00,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:00,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307700684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307700685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307700686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307700689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307700689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:00,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:00,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:00,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:00,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:00,761 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:00,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:00,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:00,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:00,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,770 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:00,770 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:00,770 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,770 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:00,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:00,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:00,772 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:00,772 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:00,772 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:00,772 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:00,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:00,777 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33807b2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:00,779 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:00,779 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,780 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:00,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:00,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:00,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:00,783 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,783 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:00,787 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:00,787 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:00,787 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:00,787 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:00,787 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:00,789 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,789 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:00,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,790 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:00,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307700790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307700790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,791 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307700790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,796 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:00,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:00,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:00,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,796 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:00,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,796 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307700795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:00,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307700795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:00,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:00,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:00,912 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:00,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:00,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:00,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:00,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,913 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,914 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:00,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:00,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:00,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:00,920 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:00,920 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:00,920 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:00,920 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:00,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7451a16b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:00,925 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:00,925 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,926 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,926 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:00,927 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:00,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:00,933 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:00,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:00,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:00,936 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:00,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:00,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:00,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:00,936 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:00,938 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,938 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:00,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,938 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:00,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:00,939 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,941 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:00,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:00,944 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:00,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:00,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:00,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:00,945 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:00,945 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:00,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:00,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307700993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307700993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:00,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307700993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307700999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307700999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,045 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:01,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:01,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:01,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:01,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:01,048 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:01,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:01,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:01,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,050 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:01,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:01,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:01,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:01,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:01,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:01,052 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,052 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:01,052 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:01,052 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:01,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,054 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,054 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c22b54[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:01,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:01,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,057 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:01,057 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,057 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:01,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:01,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:01,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:01,063 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:01,063 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:01,063 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:01,063 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:01,063 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:01,065 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,065 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,065 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,067 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:01,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:01,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:01,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,072 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:01,072 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:01,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/5bbf53bab10245349028e04aa0b71c27 2023-04-12 13:54:01,172 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:01,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:01,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:01,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:01,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:01,176 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:01,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:01,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:01,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,179 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:01,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:01,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:01,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:01,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:01,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:01,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:01,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:01,183 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1545ebe5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:01,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:01,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:01,187 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,187 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:01,189 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:01,192 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:01,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:01,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:01,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:01,192 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:01,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,197 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:01,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:01,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,200 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:01,200 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:01,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307701296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307701297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307701297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:01,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:01,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:01,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307701303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:01,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:01,304 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:01,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:01,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:01,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307701305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,306 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:01,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:01,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:01,308 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,308 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:01,308 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:01,308 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:01,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:01,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56837b68[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:01,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:01,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:01,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:01,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:01,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:01,318 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:01,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:01,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:01,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:01,318 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:01,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,321 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,322 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,325 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:01,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:01,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:01,326 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:01,326 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:01,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,426 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:01,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:01,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:01,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:01,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:01,430 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:01,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:01,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:01,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:01,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:01,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:01,436 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:01,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:01,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f20c22a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:01,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:01,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:01,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:01,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:01,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:01,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:01,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:01,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:01,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:01,451 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:01,451 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:01,451 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:01,451 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:01,451 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:01,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:01,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:01,456 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:01,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,469 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,469 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,469 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,469 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,469 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:01,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:01,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:01,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:01,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:01,470 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:01,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:01,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/d1bc99b3239c45a69f4f00cdcd13dc25 2023-04-12 13:54:01,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/c46cd7c26b594aa0bc13f2b1d8f244c0 2023-04-12 13:54:01,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/5bbf53bab10245349028e04aa0b71c27 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/5bbf53bab10245349028e04aa0b71c27 2023-04-12 13:54:01,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/5bbf53bab10245349028e04aa0b71c27, entries=200, sequenceid=572, filesize=14.3 K 2023-04-12 13:54:01,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/d1bc99b3239c45a69f4f00cdcd13dc25 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d1bc99b3239c45a69f4f00cdcd13dc25 2023-04-12 13:54:01,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:01,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:01,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d1bc99b3239c45a69f4f00cdcd13dc25, entries=150, sequenceid=572, filesize=11.9 K 2023-04-12 13:54:01,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:01,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/c46cd7c26b594aa0bc13f2b1d8f244c0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c46cd7c26b594aa0bc13f2b1d8f244c0 2023-04-12 13:54:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:01,586 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:01,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:01,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:01,587 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,587 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c46cd7c26b594aa0bc13f2b1d8f244c0, entries=150, sequenceid=572, filesize=11.9 K 2023-04-12 13:54:01,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 61f538e965579a8c5df754be9b63fe6f in 916ms, sequenceid=572, compaction requested=true 2023-04-12 13:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:01,590 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:01,590 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:01,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:01,592 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64256 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:01,592 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:54:01,592 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,592 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/173470ed3d7f4275a241341bc1b4a059, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0c56140ce5d425a9e9fbaa982ff645c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/93d1c7fb7cce4c1d8aeb25f8d15667ab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/0ce68d4dba85493e8a62592e9e1c36f9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/5bbf53bab10245349028e04aa0b71c27] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=62.8 K 2023-04-12 13:54:01,592 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,592 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:01,593 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:54:01,593 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,593 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5eab54ed20504fe8980345385d687a54, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9e0ce3b29a1248629b44e1cf686bfb3a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/3f49f562150b4d92a06ebcc555ea4f8c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/18982788da5d44b590faa014b9fd9e89, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d1bc99b3239c45a69f4f00cdcd13dc25] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=60.4 K 2023-04-12 13:54:01,592 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:01,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:01,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:01,593 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,593 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:01,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:01,596 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 173470ed3d7f4275a241341bc1b4a059, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307636001 2023-04-12 13:54:01,597 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 5eab54ed20504fe8980345385d687a54, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307636001 2023-04-12 13:54:01,597 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c0c56140ce5d425a9e9fbaa982ff645c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681307636641 2023-04-12 13:54:01,597 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9e0ce3b29a1248629b44e1cf686bfb3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681307636641 2023-04-12 13:54:01,597 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 93d1c7fb7cce4c1d8aeb25f8d15667ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1681307637789 2023-04-12 13:54:01,598 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3f49f562150b4d92a06ebcc555ea4f8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1681307637789 2023-04-12 13:54:01,598 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 0ce68d4dba85493e8a62592e9e1c36f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1681307638420 2023-04-12 13:54:01,598 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 18982788da5d44b590faa014b9fd9e89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1681307638420 2023-04-12 13:54:01,599 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5bbf53bab10245349028e04aa0b71c27, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681307640562 2023-04-12 13:54:01,599 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d1bc99b3239c45a69f4f00cdcd13dc25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681307640563 2023-04-12 13:54:01,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:01,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:01,610 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:01,611 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:01,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:01,611 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:01,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:54:01,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:01,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:54:01,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:01,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:54:01,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:01,636 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#93 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:01,640 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#95 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:01,691 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/a1e492c8612841d7b7542e9464fd7514 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a1e492c8612841d7b7542e9464fd7514 2023-04-12 13:54:01,693 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/5063eaa3b2b94d8184c29f9d0cd10e60 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5063eaa3b2b94d8184c29f9d0cd10e60 2023-04-12 13:54:01,699 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into 5063eaa3b2b94d8184c29f9d0cd10e60(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:01,699 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:01,700 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=11, startTime=1681307641590; duration=0sec 2023-04-12 13:54:01,700 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into a1e492c8612841d7b7542e9464fd7514(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:01,700 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:01,700 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:01,700 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:01,700 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=11, startTime=1681307641590; duration=0sec 2023-04-12 13:54:01,700 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:01,702 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:01,702 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:54:01,702 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:01,703 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/058649ccd27f44eaa53e76d3d4570a7f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/27dd61e0fe0240848321f40a3c08c384, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b62fd472846b46f4942fbf22a5a88157, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a694cc9466a14c3ebf359d0d30db4f3d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c46cd7c26b594aa0bc13f2b1d8f244c0] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=60.4 K 2023-04-12 13:54:01,703 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 058649ccd27f44eaa53e76d3d4570a7f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307636001 2023-04-12 13:54:01,703 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 27dd61e0fe0240848321f40a3c08c384, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1681307636641 2023-04-12 13:54:01,704 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b62fd472846b46f4942fbf22a5a88157, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1681307637789 2023-04-12 13:54:01,704 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a694cc9466a14c3ebf359d0d30db4f3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=552, earliestPutTs=1681307638420 2023-04-12 13:54:01,705 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c46cd7c26b594aa0bc13f2b1d8f244c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681307640563 2023-04-12 13:54:01,720 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:01,768 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/0a87c72cdf7948f6b7bd9093881d8a6e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0a87c72cdf7948f6b7bd9093881d8a6e 2023-04-12 13:54:01,778 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 0a87c72cdf7948f6b7bd9093881d8a6e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:01,778 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:01,778 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=11, startTime=1681307641590; duration=0sec 2023-04-12 13:54:01,778 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:01,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:01,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:01,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307701814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307701814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307701816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307701817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307701817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307701918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307701919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307701919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307701920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:01,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:01,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307701920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,067 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/58c526322d674773acb71421a6847b59 2023-04-12 13:54:02,084 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/21f09ede28de45b190d372d8d76b2b54 2023-04-12 13:54:02,097 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/0ca7136aba924a6ca0e0e85a92e37742 2023-04-12 13:54:02,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/58c526322d674773acb71421a6847b59 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/58c526322d674773acb71421a6847b59 2023-04-12 13:54:02,110 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/58c526322d674773acb71421a6847b59, entries=150, sequenceid=585, filesize=11.9 K 2023-04-12 13:54:02,110 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/21f09ede28de45b190d372d8d76b2b54 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21f09ede28de45b190d372d8d76b2b54 2023-04-12 13:54:02,115 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21f09ede28de45b190d372d8d76b2b54, entries=150, sequenceid=585, filesize=11.9 K 2023-04-12 13:54:02,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/0ca7136aba924a6ca0e0e85a92e37742 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0ca7136aba924a6ca0e0e85a92e37742 2023-04-12 13:54:02,121 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0ca7136aba924a6ca0e0e85a92e37742, entries=150, sequenceid=585, filesize=11.9 K 2023-04-12 13:54:02,122 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 61f538e965579a8c5df754be9b63fe6f in 511ms, sequenceid=585, compaction requested=false 2023-04-12 13:54:02,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:02,122 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:02,124 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:54:02,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:54:02,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:02,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:54:02,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:02,125 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:02,125 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:02,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:54:02,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:02,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:02,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611d411[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:02,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:02,128 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,131 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:02,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:02,132 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:02,132 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,132 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:02,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:02,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307702129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:02,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:02,135 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:02,135 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:02,135 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:02,135 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:02,135 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:02,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307702132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307702132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307702133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307702133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,139 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,139 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:02,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,139 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:02,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:02,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:02,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,146 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,146 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:02,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:02,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:02,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:02,146 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:02,146 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:02,147 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:02,147 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:02,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:02,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:02,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307702234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307702237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307702237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307702237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307702237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,247 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:02,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:02,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:02,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:02,250 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:02,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:02,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:02,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:02,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,252 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:02,252 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:02,252 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,252 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:02,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:02,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:02,253 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:02,253 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:02,253 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:02,254 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:02,254 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. started... 2023-04-12 13:54:02,254 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. as already flushing 2023-04-12 13:54:02,254 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:02,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:02,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,255 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:02,256 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,256 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:02,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:02,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c311df5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:02,258 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,260 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:02,260 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,260 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:02,261 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:02,262 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:02,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:02,264 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:02,264 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:02,264 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:02,264 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:02,264 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:02,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:02,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:02,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:02,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,267 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,269 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,269 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:02,269 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,269 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:02,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:02,269 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:02,269 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:02,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:02,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:02,270 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,270 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,270 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:02,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:02,270 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:02,270 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:02,270 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:02,343 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:58527 2023-04-12 13:54:02,343 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:02,348 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:58527 2023-04-12 13:54:02,348 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:02,352 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:58527 2023-04-12 13:54:02,352 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:02,353 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:58527 2023-04-12 13:54:02,353 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:02,370 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:02,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:02,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307702439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307702439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307702441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307702441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307702442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/ff188c5dab404a1091022eb1d1fe00a3 2023-04-12 13:54:02,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307702741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307702742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307702742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307702743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:02,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307702744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:02,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/4c6e581a9abc4b3299313b60b414e69a 2023-04-12 13:54:03,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:03,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:03,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55666 deadline: 1681307703245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:03,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55648 deadline: 1681307703245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:03,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:03,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55668 deadline: 1681307703248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:03,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:03,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55702 deadline: 1681307703249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:03,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:03,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55644 deadline: 1681307703249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:03,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/1c7df3a7bb884687b3ca666318972b91 2023-04-12 13:54:03,369 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/ff188c5dab404a1091022eb1d1fe00a3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ff188c5dab404a1091022eb1d1fe00a3 2023-04-12 13:54:03,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ff188c5dab404a1091022eb1d1fe00a3, entries=150, sequenceid=608, filesize=11.9 K 2023-04-12 13:54:03,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/4c6e581a9abc4b3299313b60b414e69a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4c6e581a9abc4b3299313b60b414e69a 2023-04-12 13:54:03,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4c6e581a9abc4b3299313b60b414e69a, entries=150, sequenceid=608, filesize=11.9 K 2023-04-12 13:54:03,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/1c7df3a7bb884687b3ca666318972b91 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1c7df3a7bb884687b3ca666318972b91 2023-04-12 13:54:03,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1c7df3a7bb884687b3ca666318972b91, entries=150, sequenceid=608, filesize=11.9 K 2023-04-12 13:54:03,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 61f538e965579a8c5df754be9b63fe6f in 1260ms, sequenceid=608, compaction requested=true 2023-04-12 13:54:03,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:03,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:03,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:03,384 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:03,384 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:03,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:03,385 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:03,385 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:03,385 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/A is initiating minor compaction (all files) 2023-04-12 13:54:03,385 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/B is initiating minor compaction (all files) 2023-04-12 13:54:03,385 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/A in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:03,385 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/B in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:03,385 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a1e492c8612841d7b7542e9464fd7514, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/58c526322d674773acb71421a6847b59, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ff188c5dab404a1091022eb1d1fe00a3] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.7 K 2023-04-12 13:54:03,385 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5063eaa3b2b94d8184c29f9d0cd10e60, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21f09ede28de45b190d372d8d76b2b54, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4c6e581a9abc4b3299313b60b414e69a] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.7 K 2023-04-12 13:54:03,386 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a1e492c8612841d7b7542e9464fd7514, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681307640563 2023-04-12 13:54:03,386 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 5063eaa3b2b94d8184c29f9d0cd10e60, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681307640563 2023-04-12 13:54:03,386 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 58c526322d674773acb71421a6847b59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1681307640684 2023-04-12 13:54:03,386 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 21f09ede28de45b190d372d8d76b2b54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1681307640684 2023-04-12 13:54:03,386 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ff188c5dab404a1091022eb1d1fe00a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681307641813 2023-04-12 13:54:03,387 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4c6e581a9abc4b3299313b60b414e69a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681307641813 2023-04-12 13:54:03,396 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#A#compaction#102 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:03,396 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#B#compaction#103 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:03,806 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/b45b527876d643d092cf97f7ef70ba78 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/b45b527876d643d092cf97f7ef70ba78 2023-04-12 13:54:03,806 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/6037f69beb2a42278d784ae03a8b4319 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/6037f69beb2a42278d784ae03a8b4319 2023-04-12 13:54:03,811 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/B of 61f538e965579a8c5df754be9b63fe6f into b45b527876d643d092cf97f7ef70ba78(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:03,811 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:03,811 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/B, priority=13, startTime=1681307643384; duration=0sec 2023-04-12 13:54:03,811 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/A of 61f538e965579a8c5df754be9b63fe6f into 6037f69beb2a42278d784ae03a8b4319(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:03,811 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:03,811 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:03,811 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/A, priority=13, startTime=1681307643384; duration=0sec 2023-04-12 13:54:03,811 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:03,811 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:03,812 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:03,813 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 61f538e965579a8c5df754be9b63fe6f/C is initiating minor compaction (all files) 2023-04-12 13:54:03,813 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61f538e965579a8c5df754be9b63fe6f/C in TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:03,813 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0a87c72cdf7948f6b7bd9093881d8a6e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0ca7136aba924a6ca0e0e85a92e37742, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1c7df3a7bb884687b3ca666318972b91] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp, totalSize=36.7 K 2023-04-12 13:54:03,813 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 0a87c72cdf7948f6b7bd9093881d8a6e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681307640563 2023-04-12 13:54:03,813 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 0ca7136aba924a6ca0e0e85a92e37742, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1681307640684 2023-04-12 13:54:03,814 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1c7df3a7bb884687b3ca666318972b91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1681307641813 2023-04-12 13:54:03,821 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 61f538e965579a8c5df754be9b63fe6f#C#compaction#104 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:04,231 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/894ad4d3662743c6b324ec1b3013a294 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/894ad4d3662743c6b324ec1b3013a294 2023-04-12 13:54:04,238 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61f538e965579a8c5df754be9b63fe6f/C of 61f538e965579a8c5df754be9b63fe6f into 894ad4d3662743c6b324ec1b3013a294(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:04,238 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:04,238 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f., storeName=61f538e965579a8c5df754be9b63fe6f/C, priority=13, startTime=1681307643384; duration=0sec 2023-04-12 13:54:04,238 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:04,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:04,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:04,250 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:58527 2023-04-12 13:54:04,250 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:04,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:54:04,251 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:58527 2023-04-12 13:54:04,251 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:04,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:04,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:54:04,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:04,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:54:04,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:04,253 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:58527 2023-04-12 13:54:04,253 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:04,255 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:58527 2023-04-12 13:54:04,255 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:04,256 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:58527 2023-04-12 13:54:04,256 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:04,256 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 13:54:04,256 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 200 2023-04-12 13:54:04,256 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-12 13:54:04,256 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-04-12 13:54:04,256 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-04-12 13:54:04,256 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4089 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4062 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1817 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5451 rows 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1788 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5363 rows 2023-04-12 13:54:04,257 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:54:04,257 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:58527 2023-04-12 13:54:04,257 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:04,262 INFO [Listener at localhost/46495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 13:54:04,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 13:54:04,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:04,276 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307644276"}]},"ts":"1681307644276"} 2023-04-12 13:54:04,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:04,277 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 13:54:04,279 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 13:54:04,280 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, UNASSIGN}] 2023-04-12 13:54:04,281 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, UNASSIGN 2023-04-12 13:54:04,282 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=61f538e965579a8c5df754be9b63fe6f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:04,282 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307644282"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307644282"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307644282"}]},"ts":"1681307644282"} 2023-04-12 13:54:04,283 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:54:04,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:04,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:04,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 61f538e965579a8c5df754be9b63fe6f, disabling compactions & flushes 2023-04-12 13:54:04,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:04,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:04,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/41027dd61517482eac9f460637af97d0 2023-04-12 13:54:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:05,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/04785531a8c0486098b5c48106abf779 2023-04-12 13:54:05,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:05,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/ef64dec915c8416da9203246aaf2d351 2023-04-12 13:54:05,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/41027dd61517482eac9f460637af97d0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/41027dd61517482eac9f460637af97d0 2023-04-12 13:54:05,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/41027dd61517482eac9f460637af97d0, entries=150, sequenceid=625, filesize=11.9 K 2023-04-12 13:54:05,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/04785531a8c0486098b5c48106abf779 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/04785531a8c0486098b5c48106abf779 2023-04-12 13:54:05,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/04785531a8c0486098b5c48106abf779, entries=150, sequenceid=625, filesize=11.9 K 2023-04-12 13:54:05,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/ef64dec915c8416da9203246aaf2d351 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef64dec915c8416da9203246aaf2d351 2023-04-12 13:54:05,504 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef64dec915c8416da9203246aaf2d351, entries=150, sequenceid=625, filesize=11.9 K 2023-04-12 13:54:05,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for 61f538e965579a8c5df754be9b63fe6f in 1255ms, sequenceid=625, compaction requested=false 2023-04-12 13:54:05,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:05,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:05,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:05,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. after waiting 0 ms 2023-04-12 13:54:05,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:05,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 61f538e965579a8c5df754be9b63fe6f 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-12 13:54:05,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=A 2023-04-12 13:54:05,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:05,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=B 2023-04-12 13:54:05,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:05,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61f538e965579a8c5df754be9b63fe6f, store=C 2023-04-12 13:54:05,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:05,918 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/930fe844714d41b1b585970794f6b27a 2023-04-12 13:54:06,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/f15357ccd7bc4a38a159c4833e7e667a 2023-04-12 13:54:06,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:06,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/6f921c727a2e4021980bea65e9f1df73 2023-04-12 13:54:06,748 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/A/930fe844714d41b1b585970794f6b27a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/930fe844714d41b1b585970794f6b27a 2023-04-12 13:54:06,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/930fe844714d41b1b585970794f6b27a, entries=100, sequenceid=631, filesize=9.5 K 2023-04-12 13:54:06,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/B/f15357ccd7bc4a38a159c4833e7e667a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f15357ccd7bc4a38a159c4833e7e667a 2023-04-12 13:54:06,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f15357ccd7bc4a38a159c4833e7e667a, entries=100, sequenceid=631, filesize=9.5 K 2023-04-12 13:54:06,759 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/.tmp/C/6f921c727a2e4021980bea65e9f1df73 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6f921c727a2e4021980bea65e9f1df73 2023-04-12 13:54:06,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6f921c727a2e4021980bea65e9f1df73, entries=100, sequenceid=631, filesize=9.5 K 2023-04-12 13:54:06,764 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 61f538e965579a8c5df754be9b63fe6f in 1259ms, sequenceid=631, compaction requested=true 2023-04-12 13:54:06,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/65e87fc79a524789b08d02cbb4cbf5ba, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8c9b7cf434574d448ce9dd2ba0297aec, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/986cc88899c34444a1e890b8cc755cd1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/caa9582f7aa0431fb7d70e38505684a7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2331f2d93b304899be5c851d343cf6bc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b87b8eab854b440fb32d06e7c0261021, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f01b9ebc06d4e6a9e3f5e3cd2331440, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/d77cbe5295f6431dab1e637ea8a4d23e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/38dec00d3ae943fbad2232778e1639c7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f5ebf07048f46a0bba0b0d010c20d8f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/318350f0bca0491486f37ba5abf042b1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2627ae30882c4b46a6bc8f661914abe1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/7575df5a632d4f3ea318d6be793fac3a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/f04de0d6b9a542d7b1f7023098e87e8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/51878aa47333431da511615f56599dac, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8944134af2034a5281896f7c769462af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4e666144cee64e619593cd5a72703b94, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b9dea06c947d4b23a78e0b07fabdec91, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/70196cf78a42496798c676c388811efd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ab0f8524f7c54d4597969d8f14c7f62f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0bd3cca2f6d4a53bb869d433f820d2e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/bc6c0c7bd9b745cf9d0f6510c1afcc93, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/9a66cf714d914330ba87a38ea4879c36, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a77dacd9b5034dc4b8c2e10c0d776c44, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/efdf2644c2c7494489036e5fb08dd0b8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/598fbb62ca0f48f28e9f4826b9281bcd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/173470ed3d7f4275a241341bc1b4a059, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0c56140ce5d425a9e9fbaa982ff645c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/93d1c7fb7cce4c1d8aeb25f8d15667ab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/0ce68d4dba85493e8a62592e9e1c36f9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/5bbf53bab10245349028e04aa0b71c27, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a1e492c8612841d7b7542e9464fd7514, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/58c526322d674773acb71421a6847b59, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ff188c5dab404a1091022eb1d1fe00a3] to archive 2023-04-12 13:54:06,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:54:06,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/65e87fc79a524789b08d02cbb4cbf5ba to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/65e87fc79a524789b08d02cbb4cbf5ba 2023-04-12 13:54:06,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8c9b7cf434574d448ce9dd2ba0297aec to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8c9b7cf434574d448ce9dd2ba0297aec 2023-04-12 13:54:06,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/986cc88899c34444a1e890b8cc755cd1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/986cc88899c34444a1e890b8cc755cd1 2023-04-12 13:54:06,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/caa9582f7aa0431fb7d70e38505684a7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/caa9582f7aa0431fb7d70e38505684a7 2023-04-12 13:54:06,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2331f2d93b304899be5c851d343cf6bc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2331f2d93b304899be5c851d343cf6bc 2023-04-12 13:54:06,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b87b8eab854b440fb32d06e7c0261021 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b87b8eab854b440fb32d06e7c0261021 2023-04-12 13:54:06,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f01b9ebc06d4e6a9e3f5e3cd2331440 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f01b9ebc06d4e6a9e3f5e3cd2331440 2023-04-12 13:54:06,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/d77cbe5295f6431dab1e637ea8a4d23e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/d77cbe5295f6431dab1e637ea8a4d23e 2023-04-12 13:54:06,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/38dec00d3ae943fbad2232778e1639c7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/38dec00d3ae943fbad2232778e1639c7 2023-04-12 13:54:06,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f5ebf07048f46a0bba0b0d010c20d8f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4f5ebf07048f46a0bba0b0d010c20d8f 2023-04-12 13:54:06,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/318350f0bca0491486f37ba5abf042b1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/318350f0bca0491486f37ba5abf042b1 2023-04-12 13:54:06,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2627ae30882c4b46a6bc8f661914abe1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/2627ae30882c4b46a6bc8f661914abe1 2023-04-12 13:54:06,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/7575df5a632d4f3ea318d6be793fac3a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/7575df5a632d4f3ea318d6be793fac3a 2023-04-12 13:54:06,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/f04de0d6b9a542d7b1f7023098e87e8d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/f04de0d6b9a542d7b1f7023098e87e8d 2023-04-12 13:54:06,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/51878aa47333431da511615f56599dac to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/51878aa47333431da511615f56599dac 2023-04-12 13:54:06,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8944134af2034a5281896f7c769462af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/8944134af2034a5281896f7c769462af 2023-04-12 13:54:06,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4e666144cee64e619593cd5a72703b94 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/4e666144cee64e619593cd5a72703b94 2023-04-12 13:54:06,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b9dea06c947d4b23a78e0b07fabdec91 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/b9dea06c947d4b23a78e0b07fabdec91 2023-04-12 13:54:06,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/70196cf78a42496798c676c388811efd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/70196cf78a42496798c676c388811efd 2023-04-12 13:54:06,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ab0f8524f7c54d4597969d8f14c7f62f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ab0f8524f7c54d4597969d8f14c7f62f 2023-04-12 13:54:06,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0bd3cca2f6d4a53bb869d433f820d2e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0bd3cca2f6d4a53bb869d433f820d2e 2023-04-12 13:54:06,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/bc6c0c7bd9b745cf9d0f6510c1afcc93 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/bc6c0c7bd9b745cf9d0f6510c1afcc93 2023-04-12 13:54:06,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/9a66cf714d914330ba87a38ea4879c36 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/9a66cf714d914330ba87a38ea4879c36 2023-04-12 13:54:06,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a77dacd9b5034dc4b8c2e10c0d776c44 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a77dacd9b5034dc4b8c2e10c0d776c44 2023-04-12 13:54:06,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/efdf2644c2c7494489036e5fb08dd0b8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/efdf2644c2c7494489036e5fb08dd0b8 2023-04-12 13:54:06,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/598fbb62ca0f48f28e9f4826b9281bcd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/598fbb62ca0f48f28e9f4826b9281bcd 2023-04-12 13:54:06,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/173470ed3d7f4275a241341bc1b4a059 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/173470ed3d7f4275a241341bc1b4a059 2023-04-12 13:54:06,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0c56140ce5d425a9e9fbaa982ff645c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/c0c56140ce5d425a9e9fbaa982ff645c 2023-04-12 13:54:06,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/93d1c7fb7cce4c1d8aeb25f8d15667ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/93d1c7fb7cce4c1d8aeb25f8d15667ab 2023-04-12 13:54:06,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/0ce68d4dba85493e8a62592e9e1c36f9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/0ce68d4dba85493e8a62592e9e1c36f9 2023-04-12 13:54:06,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/5bbf53bab10245349028e04aa0b71c27 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/5bbf53bab10245349028e04aa0b71c27 2023-04-12 13:54:06,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a1e492c8612841d7b7542e9464fd7514 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/a1e492c8612841d7b7542e9464fd7514 2023-04-12 13:54:06,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/58c526322d674773acb71421a6847b59 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/58c526322d674773acb71421a6847b59 2023-04-12 13:54:06,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ff188c5dab404a1091022eb1d1fe00a3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/ff188c5dab404a1091022eb1d1fe00a3 2023-04-12 13:54:06,858 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/8d7e6bfd06ee4e58b09eeb68197fb63b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a12a4ef1f40946f0986d26cf780a261b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/00401b40776d473ead34e0a2c071b991, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9f07fa1404f749a08e6dfec25a2d8839, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/630cb0368cd84417b7c4211c896bf14e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/37fe4113fde7455f858baa0522b66c99, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f56864acd85940dba3932ad030515ca6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/0be9ad9f1db8491cb3ffdaafa6e4a772, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/c9d4c41faf7f4f3f82251fb8d90741d2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a5ce04b7f13044988467b882f78bdae1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e1e064fb81144ca5950b3ad008783467, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e38f8f83415c42cab187d28daa056184, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/122226e6995d4ca18e52c25b4b169bb5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21405e8f3ac247a3af0929f1777849b4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/77514818806f4e0680b71992e510b47f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/7a95cb4b0e2b4574b80abf0a86af1d0d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4713e536cc1d4829a7847a9d65e9c718, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/184357bae3f04fc8baf6271b82541793, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a219bf818e05481f841e49fdac632e9f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/fe758fe19c3f4c5989fda2b8ab65177c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/01f3efceeda742eca1a2720f4ee57fae, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/248f70215109450a881d8aee549802a6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/682a14ca91114402852e3f872ac8e105, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d165fa14bbc049febfcaa31a0b54fcd9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/97793627139b427390aef7b44e8b88b8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5eab54ed20504fe8980345385d687a54, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4076a3ffaf434990859e58ce7546f831, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9e0ce3b29a1248629b44e1cf686bfb3a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/3f49f562150b4d92a06ebcc555ea4f8c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/18982788da5d44b590faa014b9fd9e89, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5063eaa3b2b94d8184c29f9d0cd10e60, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d1bc99b3239c45a69f4f00cdcd13dc25, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21f09ede28de45b190d372d8d76b2b54, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4c6e581a9abc4b3299313b60b414e69a] to archive 2023-04-12 13:54:06,859 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:54:06,861 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/8d7e6bfd06ee4e58b09eeb68197fb63b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/8d7e6bfd06ee4e58b09eeb68197fb63b 2023-04-12 13:54:06,862 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a12a4ef1f40946f0986d26cf780a261b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a12a4ef1f40946f0986d26cf780a261b 2023-04-12 13:54:06,863 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/00401b40776d473ead34e0a2c071b991 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/00401b40776d473ead34e0a2c071b991 2023-04-12 13:54:06,864 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9f07fa1404f749a08e6dfec25a2d8839 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9f07fa1404f749a08e6dfec25a2d8839 2023-04-12 13:54:06,865 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/630cb0368cd84417b7c4211c896bf14e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/630cb0368cd84417b7c4211c896bf14e 2023-04-12 13:54:06,866 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/37fe4113fde7455f858baa0522b66c99 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/37fe4113fde7455f858baa0522b66c99 2023-04-12 13:54:06,867 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f56864acd85940dba3932ad030515ca6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f56864acd85940dba3932ad030515ca6 2023-04-12 13:54:06,868 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/0be9ad9f1db8491cb3ffdaafa6e4a772 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/0be9ad9f1db8491cb3ffdaafa6e4a772 2023-04-12 13:54:06,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/c9d4c41faf7f4f3f82251fb8d90741d2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/c9d4c41faf7f4f3f82251fb8d90741d2 2023-04-12 13:54:06,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a5ce04b7f13044988467b882f78bdae1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a5ce04b7f13044988467b882f78bdae1 2023-04-12 13:54:06,871 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e1e064fb81144ca5950b3ad008783467 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e1e064fb81144ca5950b3ad008783467 2023-04-12 13:54:06,872 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e38f8f83415c42cab187d28daa056184 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/e38f8f83415c42cab187d28daa056184 2023-04-12 13:54:06,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/122226e6995d4ca18e52c25b4b169bb5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/122226e6995d4ca18e52c25b4b169bb5 2023-04-12 13:54:06,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21405e8f3ac247a3af0929f1777849b4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21405e8f3ac247a3af0929f1777849b4 2023-04-12 13:54:06,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/77514818806f4e0680b71992e510b47f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/77514818806f4e0680b71992e510b47f 2023-04-12 13:54:06,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/7a95cb4b0e2b4574b80abf0a86af1d0d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/7a95cb4b0e2b4574b80abf0a86af1d0d 2023-04-12 13:54:06,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4713e536cc1d4829a7847a9d65e9c718 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4713e536cc1d4829a7847a9d65e9c718 2023-04-12 13:54:06,878 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/184357bae3f04fc8baf6271b82541793 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/184357bae3f04fc8baf6271b82541793 2023-04-12 13:54:06,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a219bf818e05481f841e49fdac632e9f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/a219bf818e05481f841e49fdac632e9f 2023-04-12 13:54:06,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/fe758fe19c3f4c5989fda2b8ab65177c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/fe758fe19c3f4c5989fda2b8ab65177c 2023-04-12 13:54:06,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/01f3efceeda742eca1a2720f4ee57fae to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/01f3efceeda742eca1a2720f4ee57fae 2023-04-12 13:54:06,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/248f70215109450a881d8aee549802a6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/248f70215109450a881d8aee549802a6 2023-04-12 13:54:06,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/682a14ca91114402852e3f872ac8e105 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/682a14ca91114402852e3f872ac8e105 2023-04-12 13:54:06,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d165fa14bbc049febfcaa31a0b54fcd9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d165fa14bbc049febfcaa31a0b54fcd9 2023-04-12 13:54:06,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/97793627139b427390aef7b44e8b88b8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/97793627139b427390aef7b44e8b88b8 2023-04-12 13:54:06,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5eab54ed20504fe8980345385d687a54 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5eab54ed20504fe8980345385d687a54 2023-04-12 13:54:06,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4076a3ffaf434990859e58ce7546f831 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4076a3ffaf434990859e58ce7546f831 2023-04-12 13:54:06,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9e0ce3b29a1248629b44e1cf686bfb3a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/9e0ce3b29a1248629b44e1cf686bfb3a 2023-04-12 13:54:06,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/3f49f562150b4d92a06ebcc555ea4f8c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/3f49f562150b4d92a06ebcc555ea4f8c 2023-04-12 13:54:06,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/18982788da5d44b590faa014b9fd9e89 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/18982788da5d44b590faa014b9fd9e89 2023-04-12 13:54:06,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5063eaa3b2b94d8184c29f9d0cd10e60 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/5063eaa3b2b94d8184c29f9d0cd10e60 2023-04-12 13:54:06,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d1bc99b3239c45a69f4f00cdcd13dc25 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/d1bc99b3239c45a69f4f00cdcd13dc25 2023-04-12 13:54:06,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21f09ede28de45b190d372d8d76b2b54 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/21f09ede28de45b190d372d8d76b2b54 2023-04-12 13:54:06,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4c6e581a9abc4b3299313b60b414e69a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/4c6e581a9abc4b3299313b60b414e69a 2023-04-12 13:54:06,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c1f32eacd9204ba88b1fe2f8d1282ad1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/acffe2a0705346fdacf53f988b4a5b26, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/f06bbe90e10143f09d48de0f8b396dbd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/48ddb73826094bb3bc50ee9c387e0f70, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7f3daffd2da14bf0901e1da3f7a90589, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/28516c23bb3a41dca56a26d87d7a984f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/eca514388f2f410aadef1e7f01b055f6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b21258620abb4e5ba364f3bd4922678c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef3998180fa2432ebab452c79807c98c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/84b1d11dd05246699e54151c6e4ae0fc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/82adc42f39364a94a4396a300ddd79a0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a702d5978989413aa279ce390ba6800c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1449c6bbf5bd47b78856ed7d36283fbb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/4935c4f88c484567903a546b252341f6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/8b7842d65255494482bcee3a01393720, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c493c024dea94b8e9b1f076b8af87de6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/85770e8423754fed8aeff129f8ff0c4e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/cc8abeb6e2264220a8c527879da8aff0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/46c5ab36e9644a6581f36de1cd897de1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7b6926c290994bda8ce10507842b1766, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/80352f072f62454fa0a82467313dc4f2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6ab4e95718674f0fbdf7cdf556d406cd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/79d72f139e654fd1afdaee9a6bea2832, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/372de978db6145f9953b38049adffeb2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/74334584c19c4a7e8fcc948550b36d3d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/058649ccd27f44eaa53e76d3d4570a7f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/13e92edad4a646eaa14b6e0018b49727, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/27dd61e0fe0240848321f40a3c08c384, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b62fd472846b46f4942fbf22a5a88157, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a694cc9466a14c3ebf359d0d30db4f3d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0a87c72cdf7948f6b7bd9093881d8a6e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c46cd7c26b594aa0bc13f2b1d8f244c0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0ca7136aba924a6ca0e0e85a92e37742, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1c7df3a7bb884687b3ca666318972b91] to archive 2023-04-12 13:54:06,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:54:06,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c1f32eacd9204ba88b1fe2f8d1282ad1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c1f32eacd9204ba88b1fe2f8d1282ad1 2023-04-12 13:54:06,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/acffe2a0705346fdacf53f988b4a5b26 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/acffe2a0705346fdacf53f988b4a5b26 2023-04-12 13:54:06,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/f06bbe90e10143f09d48de0f8b396dbd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/f06bbe90e10143f09d48de0f8b396dbd 2023-04-12 13:54:06,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/48ddb73826094bb3bc50ee9c387e0f70 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/48ddb73826094bb3bc50ee9c387e0f70 2023-04-12 13:54:06,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7f3daffd2da14bf0901e1da3f7a90589 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7f3daffd2da14bf0901e1da3f7a90589 2023-04-12 13:54:06,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/28516c23bb3a41dca56a26d87d7a984f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/28516c23bb3a41dca56a26d87d7a984f 2023-04-12 13:54:06,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/eca514388f2f410aadef1e7f01b055f6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/eca514388f2f410aadef1e7f01b055f6 2023-04-12 13:54:06,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b21258620abb4e5ba364f3bd4922678c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b21258620abb4e5ba364f3bd4922678c 2023-04-12 13:54:06,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef3998180fa2432ebab452c79807c98c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef3998180fa2432ebab452c79807c98c 2023-04-12 13:54:06,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/84b1d11dd05246699e54151c6e4ae0fc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/84b1d11dd05246699e54151c6e4ae0fc 2023-04-12 13:54:06,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/82adc42f39364a94a4396a300ddd79a0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/82adc42f39364a94a4396a300ddd79a0 2023-04-12 13:54:06,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a702d5978989413aa279ce390ba6800c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a702d5978989413aa279ce390ba6800c 2023-04-12 13:54:06,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1449c6bbf5bd47b78856ed7d36283fbb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1449c6bbf5bd47b78856ed7d36283fbb 2023-04-12 13:54:06,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/4935c4f88c484567903a546b252341f6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/4935c4f88c484567903a546b252341f6 2023-04-12 13:54:06,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/8b7842d65255494482bcee3a01393720 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/8b7842d65255494482bcee3a01393720 2023-04-12 13:54:06,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c493c024dea94b8e9b1f076b8af87de6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c493c024dea94b8e9b1f076b8af87de6 2023-04-12 13:54:06,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/85770e8423754fed8aeff129f8ff0c4e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/85770e8423754fed8aeff129f8ff0c4e 2023-04-12 13:54:06,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/cc8abeb6e2264220a8c527879da8aff0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/cc8abeb6e2264220a8c527879da8aff0 2023-04-12 13:54:06,919 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/46c5ab36e9644a6581f36de1cd897de1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/46c5ab36e9644a6581f36de1cd897de1 2023-04-12 13:54:06,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7b6926c290994bda8ce10507842b1766 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/7b6926c290994bda8ce10507842b1766 2023-04-12 13:54:06,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/80352f072f62454fa0a82467313dc4f2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/80352f072f62454fa0a82467313dc4f2 2023-04-12 13:54:06,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6ab4e95718674f0fbdf7cdf556d406cd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6ab4e95718674f0fbdf7cdf556d406cd 2023-04-12 13:54:06,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/79d72f139e654fd1afdaee9a6bea2832 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/79d72f139e654fd1afdaee9a6bea2832 2023-04-12 13:54:06,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/372de978db6145f9953b38049adffeb2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/372de978db6145f9953b38049adffeb2 2023-04-12 13:54:06,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/74334584c19c4a7e8fcc948550b36d3d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/74334584c19c4a7e8fcc948550b36d3d 2023-04-12 13:54:06,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/058649ccd27f44eaa53e76d3d4570a7f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/058649ccd27f44eaa53e76d3d4570a7f 2023-04-12 13:54:06,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/13e92edad4a646eaa14b6e0018b49727 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/13e92edad4a646eaa14b6e0018b49727 2023-04-12 13:54:06,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/27dd61e0fe0240848321f40a3c08c384 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/27dd61e0fe0240848321f40a3c08c384 2023-04-12 13:54:06,929 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b62fd472846b46f4942fbf22a5a88157 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/b62fd472846b46f4942fbf22a5a88157 2023-04-12 13:54:06,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a694cc9466a14c3ebf359d0d30db4f3d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/a694cc9466a14c3ebf359d0d30db4f3d 2023-04-12 13:54:06,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0a87c72cdf7948f6b7bd9093881d8a6e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0a87c72cdf7948f6b7bd9093881d8a6e 2023-04-12 13:54:06,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c46cd7c26b594aa0bc13f2b1d8f244c0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/c46cd7c26b594aa0bc13f2b1d8f244c0 2023-04-12 13:54:06,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0ca7136aba924a6ca0e0e85a92e37742 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/0ca7136aba924a6ca0e0e85a92e37742 2023-04-12 13:54:06,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1c7df3a7bb884687b3ca666318972b91 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/1c7df3a7bb884687b3ca666318972b91 2023-04-12 13:54:06,938 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/recovered.edits/634.seqid, newMaxSeqId=634, maxSeqId=1 2023-04-12 13:54:06,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f. 2023-04-12 13:54:06,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 61f538e965579a8c5df754be9b63fe6f: 2023-04-12 13:54:06,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:06,942 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=61f538e965579a8c5df754be9b63fe6f, regionState=CLOSED 2023-04-12 13:54:06,942 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307646942"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307646942"}]},"ts":"1681307646942"} 2023-04-12 13:54:06,945 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-12 13:54:06,945 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 61f538e965579a8c5df754be9b63fe6f, server=jenkins-hbase4.apache.org,35663,1681307616979 in 2.6610 sec 2023-04-12 13:54:06,947 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-12 13:54:06,947 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61f538e965579a8c5df754be9b63fe6f, UNASSIGN in 2.6650 sec 2023-04-12 13:54:06,948 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307646947"}]},"ts":"1681307646947"} 2023-04-12 13:54:06,948 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 13:54:06,950 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 13:54:06,951 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6820 sec 2023-04-12 13:54:08,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-12 13:54:08,383 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-12 13:54:08,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 13:54:08,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,395 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,397 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-12 13:54:08,401 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:08,404 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/recovered.edits] 2023-04-12 13:54:08,407 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/41027dd61517482eac9f460637af97d0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/41027dd61517482eac9f460637af97d0 2023-04-12 13:54:08,409 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/6037f69beb2a42278d784ae03a8b4319 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/6037f69beb2a42278d784ae03a8b4319 2023-04-12 13:54:08,410 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/930fe844714d41b1b585970794f6b27a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/A/930fe844714d41b1b585970794f6b27a 2023-04-12 13:54:08,413 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/04785531a8c0486098b5c48106abf779 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/04785531a8c0486098b5c48106abf779 2023-04-12 13:54:08,414 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/b45b527876d643d092cf97f7ef70ba78 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/b45b527876d643d092cf97f7ef70ba78 2023-04-12 13:54:08,415 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f15357ccd7bc4a38a159c4833e7e667a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/B/f15357ccd7bc4a38a159c4833e7e667a 2023-04-12 13:54:08,418 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6f921c727a2e4021980bea65e9f1df73 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/6f921c727a2e4021980bea65e9f1df73 2023-04-12 13:54:08,419 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/894ad4d3662743c6b324ec1b3013a294 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/894ad4d3662743c6b324ec1b3013a294 2023-04-12 13:54:08,420 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef64dec915c8416da9203246aaf2d351 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/C/ef64dec915c8416da9203246aaf2d351 2023-04-12 13:54:08,423 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/recovered.edits/634.seqid to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f/recovered.edits/634.seqid 2023-04-12 13:54:08,424 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/61f538e965579a8c5df754be9b63fe6f 2023-04-12 13:54:08,424 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:54:08,427 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,448 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 13:54:08,491 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 13:54:08,492 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,492 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 13:54:08,493 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681307648493"}]},"ts":"9223372036854775807"} 2023-04-12 13:54:08,495 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 13:54:08,495 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 61f538e965579a8c5df754be9b63fe6f, NAME => 'TestAcidGuarantees,,1681307621064.61f538e965579a8c5df754be9b63fe6f.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 13:54:08,495 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 13:54:08,495 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681307648495"}]},"ts":"9223372036854775807"} 2023-04-12 13:54:08,497 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 13:54:08,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-12 13:54:08,500 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,501 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 112 msec 2023-04-12 13:54:08,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-12 13:54:08,700 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-12 13:54:08,717 INFO [Listener at localhost/46495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:35663-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7de1e028-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-419731489_15 at /127.0.0.1:45520 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7de1e028-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-419731489_15 at /127.0.0.1:45574 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7de1e028-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x7de1e028-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=563 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=451 (was 355) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=522 (was 1221) 2023-04-12 13:54:08,729 INFO [Listener at localhost/46495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=451, ProcessCount=169, AvailableMemoryMB=521 2023-04-12 13:54:08,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:54:08,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:54:08,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:08,735 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:54:08,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-12 13:54:08,736 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:54:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 13:54:08,737 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:08,738 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df empty. 2023-04-12 13:54:08,738 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:08,738 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:54:08,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 13:54:09,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 13:54:09,151 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:54:09,152 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6b28deeffcdc135c24d47af5464730df, NAME => 'TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:54:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 13:54:09,559 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:54:09,559 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6b28deeffcdc135c24d47af5464730df, disabling compactions & flushes 2023-04-12 13:54:09,559 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,559 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,559 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. after waiting 0 ms 2023-04-12 13:54:09,559 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,559 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,559 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:09,562 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:54:09,563 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307649563"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307649563"}]},"ts":"1681307649563"} 2023-04-12 13:54:09,565 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:54:09,566 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:54:09,567 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307649566"}]},"ts":"1681307649566"} 2023-04-12 13:54:09,568 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 13:54:09,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, ASSIGN}] 2023-04-12 13:54:09,574 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, ASSIGN 2023-04-12 13:54:09,575 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:54:09,726 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:09,727 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307649726"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307649726"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307649726"}]},"ts":"1681307649726"} 2023-04-12 13:54:09,729 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:54:09,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 13:54:09,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6b28deeffcdc135c24d47af5464730df, NAME => 'TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:54:09,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:54:09,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,887 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,888 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A 2023-04-12 13:54:09,889 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A 2023-04-12 13:54:09,889 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:09,889 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b28deeffcdc135c24d47af5464730df columnFamilyName A 2023-04-12 13:54:09,889 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(310): Store=6b28deeffcdc135c24d47af5464730df/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:09,890 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,891 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B 2023-04-12 13:54:09,891 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B 2023-04-12 13:54:09,891 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:09,891 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b28deeffcdc135c24d47af5464730df columnFamilyName B 2023-04-12 13:54:09,892 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(310): Store=6b28deeffcdc135c24d47af5464730df/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:09,892 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,893 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C 2023-04-12 13:54:09,893 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C 2023-04-12 13:54:09,893 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:09,893 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b28deeffcdc135c24d47af5464730df columnFamilyName C 2023-04-12 13:54:09,893 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(310): Store=6b28deeffcdc135c24d47af5464730df/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:09,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:54:09,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:09,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:54:09,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6b28deeffcdc135c24d47af5464730df; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65139169, jitterRate=-0.02935074269771576}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:54:09,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:09,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., pid=18, masterSystemTime=1681307649881 2023-04-12 13:54:09,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:09,903 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:09,903 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307649903"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307649903"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307649903"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307649903"}]},"ts":"1681307649903"} 2023-04-12 13:54:09,906 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-12 13:54:09,906 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 in 175 msec 2023-04-12 13:54:09,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-12 13:54:09,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, ASSIGN in 334 msec 2023-04-12 13:54:09,908 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:54:09,908 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307649908"}]},"ts":"1681307649908"} 2023-04-12 13:54:09,909 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 13:54:09,911 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:54:09,913 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1780 sec 2023-04-12 13:54:10,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-12 13:54:10,842 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-12 13:54:10,843 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:10,862 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:10,866 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:10,872 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36094, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:10,873 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:54:10,876 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:54:10,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:54:10,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:54:10,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:11,316 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 13:54:11,317 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:54:11,317 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 13:54:11,322 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-12 13:54:11,337 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, REOPEN/MOVE}] 2023-04-12 13:54:11,338 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, REOPEN/MOVE 2023-04-12 13:54:11,339 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,339 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307651339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307651339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307651339"}]},"ts":"1681307651339"} 2023-04-12 13:54:11,341 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:54:11,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,494 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6b28deeffcdc135c24d47af5464730df, disabling compactions & flushes 2023-04-12 13:54:11,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. after waiting 0 ms 2023-04-12 13:54:11,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 13:54:11,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:11,499 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 6b28deeffcdc135c24d47af5464730df to self. 2023-04-12 13:54:11,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,502 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=CLOSED 2023-04-12 13:54:11,502 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307651502"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307651502"}]},"ts":"1681307651502"} 2023-04-12 13:54:11,505 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-12 13:54:11,505 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 in 162 msec 2023-04-12 13:54:11,506 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=true 2023-04-12 13:54:11,657 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,657 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307651656"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307651656"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307651656"}]},"ts":"1681307651656"} 2023-04-12 13:54:11,659 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:54:11,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6b28deeffcdc135c24d47af5464730df, NAME => 'TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:54:11,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:54:11,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,814 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,817 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,818 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A 2023-04-12 13:54:11,818 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A 2023-04-12 13:54:11,818 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:11,821 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b28deeffcdc135c24d47af5464730df columnFamilyName A 2023-04-12 13:54:11,823 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(310): Store=6b28deeffcdc135c24d47af5464730df/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:11,824 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,825 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B 2023-04-12 13:54:11,825 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B 2023-04-12 13:54:11,825 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:11,825 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b28deeffcdc135c24d47af5464730df columnFamilyName B 2023-04-12 13:54:11,826 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(310): Store=6b28deeffcdc135c24d47af5464730df/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:11,826 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,827 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C 2023-04-12 13:54:11,827 DEBUG [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C 2023-04-12 13:54:11,827 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:11,828 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b28deeffcdc135c24d47af5464730df columnFamilyName C 2023-04-12 13:54:11,828 INFO [StoreOpener-6b28deeffcdc135c24d47af5464730df-1] regionserver.HStore(310): Store=6b28deeffcdc135c24d47af5464730df/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:11,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:54:11,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6b28deeffcdc135c24d47af5464730df; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75317091, jitterRate=0.12231211364269257}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:54:11,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:11,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., pid=23, masterSystemTime=1681307651810 2023-04-12 13:54:11,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,838 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,838 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307651837"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307651837"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307651837"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307651837"}]},"ts":"1681307651837"} 2023-04-12 13:54:11,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-12 13:54:11,841 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 in 181 msec 2023-04-12 13:54:11,842 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-12 13:54:11,842 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, REOPEN/MOVE in 504 msec 2023-04-12 13:54:11,844 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-12 13:54:11,845 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 522 msec 2023-04-12 13:54:11,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 958 msec 2023-04-12 13:54:11,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-12 13:54:11,856 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,860 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,861 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,865 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,865 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,869 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,869 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,875 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,876 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,879 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,879 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,883 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,884 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,887 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,887 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,892 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,892 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:11,895 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:11,906 DEBUG [hconnection-0x78420db7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,906 DEBUG [hconnection-0x6a739aeb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,907 DEBUG [hconnection-0x141ee72d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,907 DEBUG [hconnection-0x11c7dd67-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,908 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:11,909 DEBUG [hconnection-0x5d7da072-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,911 DEBUG [hconnection-0x5cdb4c18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,911 DEBUG [hconnection-0x15018fae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,911 DEBUG [hconnection-0x32f0b4b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,915 DEBUG [hconnection-0x5b9818f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:11,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,918 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36176, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,918 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:11,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:11,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:11,942 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:11,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:11,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:11,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:11,943 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,943 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:11,948 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:11,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:11,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:11,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:11,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:11,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:11,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:11,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:54:11,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:11,952 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:11,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:11,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:11,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:11,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:11,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:11,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:11,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:11,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:11,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:11,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:11,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,964 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,965 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a46920[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:11,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:11,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:11,969 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:11,969 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:11,969 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:11,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:11,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:11,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:11,977 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:11,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:11,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:11,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:11,977 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:11,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:11,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:11,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:11,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:11,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:11,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307711983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:11,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307711983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:11,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307711984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:11,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307711985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:11,985 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:11,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307711985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,985 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:11,986 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:11,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:11,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:11,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:11,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:11,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:11,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:11,986 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:11,986 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:11,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,087 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307712086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307712086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307712086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307712086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307712087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,096 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,096 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,096 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,096 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,098 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,098 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,108 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,109 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,109 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,109 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,109 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,109 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,109 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,111 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,112 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,112 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bcc4ac[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,115 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,118 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,118 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,118 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,118 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,122 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,122 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,122 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,122 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,122 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,124 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,124 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,124 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,124 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,126 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,130 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,131 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,131 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:12,131 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,231 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,240 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,240 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,240 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,240 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,241 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,241 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,242 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,243 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,243 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,253 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,253 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,254 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,264 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,264 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,264 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,264 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,264 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,264 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,265 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,265 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,265 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,275 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,275 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,275 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,276 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500522f6[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,278 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,279 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,279 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,280 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,281 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,281 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,281 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,284 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,284 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,286 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,287 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,288 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,288 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307712289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307712290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307712290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307712291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307712291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,296 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,296 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,296 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,296 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,297 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,297 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:12,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,397 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,408 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,410 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,412 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,412 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,412 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,412 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ef43b6[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,421 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,421 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,426 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,434 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041298ebbb66ad044668b607acbd237af051 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041298ebbb66ad044668b607acbd237af051 2023-04-12 13:54:12,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,439 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,440 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,447 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/2c62f366ca204f88a2dcb507b2630439 2023-04-12 13:54:12,502 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 13:54:12,540 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,548 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,551 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,558 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3650cb62[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,563 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,564 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,567 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,568 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,573 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,577 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,580 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307712594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307712594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307712595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307712595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:12,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307712600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,680 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,684 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,687 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,689 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,689 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,689 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,689 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,691 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@877f0a0[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,695 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,695 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,699 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,700 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,707 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,707 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:12,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,807 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,812 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,812 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,812 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,812 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,822 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,822 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,822 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,825 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711c61ca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,829 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,829 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,829 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,829 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,834 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,837 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,837 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,839 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,839 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,839 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,843 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,843 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:12,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,866 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/ab7a8bf15b0d4128a5b462726518b571 2023-04-12 13:54:12,943 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:12,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:12,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:12,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:12,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:12,947 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:12,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:12,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:12,948 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,948 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,950 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,950 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:12,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:12,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:12,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:12,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:12,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:12,952 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:12,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:12,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff07eac[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:12,956 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:12,956 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,957 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,958 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:12,958 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:12,958 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:12,958 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,958 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:12,959 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:12,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:12,962 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:12,962 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:12,962 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:12,962 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:12,962 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:12,963 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,963 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:12,963 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:12,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:12,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:12,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:12,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:12,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:12,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:12,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:12,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:12,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:12,970 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:12,970 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:13,070 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:13,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:13,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:13,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:13,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:13,074 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:13,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:13,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:13,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:13,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:13,076 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:13,076 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,076 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:13,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:13,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:13,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:13,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:13,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:13,078 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:13,078 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:13,078 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:13,078 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:13,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:13,080 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,080 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:13,080 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,080 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:13,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:13,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f4bb3eb[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:13,082 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:13,082 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,083 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,083 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:13,083 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,083 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:13,083 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:13,083 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:13,086 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:13,086 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:13,086 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:13,086 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:13,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:13,090 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:13,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:13,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:13,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:13,090 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:13,093 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:13,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:13,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:13,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,096 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:13,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:13,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:13,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:13,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,100 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,100 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:13,100 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:13,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:13,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307713099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:13,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:13,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307713101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307713101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:13,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307713102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:13,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:13,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307713107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,200 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:13,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:13,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:13,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:13,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:13,203 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:13,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:13,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:13,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:13,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,204 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:13,205 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:13,206 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,206 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:13,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:13,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:13,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:13,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:13,208 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:13,208 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:13,208 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:13,208 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:13,208 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:13,210 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,210 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,211 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:13,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688eba52[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:13,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,221 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:13,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:13,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:13,222 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,222 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:13,223 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:13,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:13,226 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:13,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:13,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:13,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:13,227 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:13,232 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,232 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:13,232 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:13,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:13,233 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,235 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:13,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:13,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:13,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:13,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:13,241 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,241 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:13,241 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:13,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:13,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/a8d412e0f2a64ec784292fa2263289d7 2023-04-12 13:54:13,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/2c62f366ca204f88a2dcb507b2630439 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2c62f366ca204f88a2dcb507b2630439 2023-04-12 13:54:13,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2c62f366ca204f88a2dcb507b2630439, entries=150, sequenceid=16, filesize=25.1 K 2023-04-12 13:54:13,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/ab7a8bf15b0d4128a5b462726518b571 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/ab7a8bf15b0d4128a5b462726518b571 2023-04-12 13:54:13,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/ab7a8bf15b0d4128a5b462726518b571, entries=150, sequenceid=16, filesize=11.6 K 2023-04-12 13:54:13,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/a8d412e0f2a64ec784292fa2263289d7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a8d412e0f2a64ec784292fa2263289d7 2023-04-12 13:54:13,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a8d412e0f2a64ec784292fa2263289d7, entries=150, sequenceid=16, filesize=11.6 K 2023-04-12 13:54:13,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6b28deeffcdc135c24d47af5464730df in 1352ms, sequenceid=16, compaction requested=false 2023-04-12 13:54:13,304 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 13:54:13,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:13,341 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:13,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:13,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:13,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:13,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:13,347 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:13,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:13,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:13,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:13,348 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,348 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:13,350 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,350 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:13,350 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:13,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:13,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:13,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:13,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:13,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:13,352 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:54:13,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:13,353 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:13,353 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:13,353 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:13,353 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:13,353 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:13,798 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127e1ed45e69be48579cf8eea488539621 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e1ed45e69be48579cf8eea488539621 2023-04-12 13:54:13,836 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/46d8e43d4bb34293aaf80bdec48b3fb3 2023-04-12 13:54:13,889 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/7ef84ae724c44baf9cc39a53ebf5b528 2023-04-12 13:54:14,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:14,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:14,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307714123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307714125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307714125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307714127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307714127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307714232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307714232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307714230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307714233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307714240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,335 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/67a8977a98bc4588ba389ea40daa1c67 2023-04-12 13:54:14,340 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/46d8e43d4bb34293aaf80bdec48b3fb3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/46d8e43d4bb34293aaf80bdec48b3fb3 2023-04-12 13:54:14,346 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/46d8e43d4bb34293aaf80bdec48b3fb3, entries=150, sequenceid=36, filesize=25.1 K 2023-04-12 13:54:14,347 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/7ef84ae724c44baf9cc39a53ebf5b528 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ef84ae724c44baf9cc39a53ebf5b528 2023-04-12 13:54:14,353 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ef84ae724c44baf9cc39a53ebf5b528, entries=150, sequenceid=36, filesize=11.6 K 2023-04-12 13:54:14,354 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/67a8977a98bc4588ba389ea40daa1c67 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67a8977a98bc4588ba389ea40daa1c67 2023-04-12 13:54:14,360 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67a8977a98bc4588ba389ea40daa1c67, entries=150, sequenceid=36, filesize=11.6 K 2023-04-12 13:54:14,361 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6b28deeffcdc135c24d47af5464730df in 1009ms, sequenceid=36, compaction requested=false 2023-04-12 13:54:14,362 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:14,362 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:14,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:14,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:14,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:14,365 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,365 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,365 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,365 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:14,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8e94b8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:14,369 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:14,369 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:14,371 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:14,372 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:14,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:14,373 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,373 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,373 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,373 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:14,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:14,378 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:14,378 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:14,378 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:14,378 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:14,378 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:14,380 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,380 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,380 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,381 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,383 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,387 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:14,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,387 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,387 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:14,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,387 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:14,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:14,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:14,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:14,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:14,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:14,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:14,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:14,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:14,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307714457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307714457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307714458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307714459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307714462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,477 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412fce95721f10d4243938c4408f341e573 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fce95721f10d4243938c4408f341e573 2023-04-12 13:54:14,487 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:14,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:14,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:14,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:14,501 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:14,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:14,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:14,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,502 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,504 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,504 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,504 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,504 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:14,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:14,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:14,507 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/f4a521149fd741208a0400c6b33ab3cf 2023-04-12 13:54:14,507 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,507 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:14,507 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:14,507 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:14,507 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:14,509 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,510 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:14,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e7805b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:14,511 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:14,511 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,512 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,512 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:14,513 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:14,513 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:14,513 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,513 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:14,514 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,514 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:14,514 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,514 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:14,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:14,519 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:14,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:14,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:14,519 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:14,519 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:14,522 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,522 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,522 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,522 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,524 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:14,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,526 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:14,527 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:14,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307714562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307714563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307714563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307714563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307714564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,627 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:14,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:14,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:14,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:14,635 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:14,635 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,635 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:14,636 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:14,636 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,636 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,637 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,637 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:14,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:14,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:14,639 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,639 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:14,639 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:14,639 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:14,641 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,641 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,641 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,641 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:14,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1431bd87[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:14,642 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:14,642 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,645 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:14,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:14,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:14,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,646 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,647 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:14,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:14,649 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:14,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:14,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:14,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:14,649 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:14,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,651 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,652 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,656 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:14,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:14,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,656 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:14,656 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,757 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:14,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:14,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:14,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:14,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:14,760 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:14,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:14,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:14,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,762 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,762 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:14,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307714765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307714766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307714766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307714766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:14,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307714768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:14,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:14,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:14,779 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,779 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:14,779 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:14,779 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:14,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,781 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:14,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@970c979[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:14,783 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:14,783 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,784 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:14,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:14,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:14,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:14,789 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,790 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,790 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:14,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:14,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:14,790 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:14,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:14,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:14,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:14,790 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:14,791 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,791 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,792 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,796 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:14,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:14,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,796 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:14,796 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:14,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,797 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,797 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:14,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,897 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:14,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:14,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:14,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:14,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:14,902 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:14,902 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,902 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:14,902 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:14,903 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,903 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,904 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:14,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:14,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:14,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:14,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:14,907 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,908 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:14,908 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:14,908 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:14,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:14,914 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,914 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,915 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,915 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:14,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b7bb[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:14,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:14,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,919 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:14,919 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:14,919 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:14,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,921 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:14,921 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,921 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:14,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:14,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:14,921 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:14,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:14,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:14,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:14,921 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:14,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c245e7b52b564d26bc74be17bab2c2c8 2023-04-12 13:54:14,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,924 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:14,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,925 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:14,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:14,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:14,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:14,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:14,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:14,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:14,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:14,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:14,935 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:14,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:14,936 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:14,936 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:14,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/90c839c7cbda4e1aaba7639df17b45fe 2023-04-12 13:54:14,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/f4a521149fd741208a0400c6b33ab3cf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f4a521149fd741208a0400c6b33ab3cf 2023-04-12 13:54:14,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f4a521149fd741208a0400c6b33ab3cf, entries=200, sequenceid=49, filesize=31.9 K 2023-04-12 13:54:14,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c245e7b52b564d26bc74be17bab2c2c8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c245e7b52b564d26bc74be17bab2c2c8 2023-04-12 13:54:14,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c245e7b52b564d26bc74be17bab2c2c8, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:54:14,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/90c839c7cbda4e1aaba7639df17b45fe as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/90c839c7cbda4e1aaba7639df17b45fe 2023-04-12 13:54:14,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/90c839c7cbda4e1aaba7639df17b45fe, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:54:14,982 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6b28deeffcdc135c24d47af5464730df in 544ms, sequenceid=49, compaction requested=true 2023-04-12 13:54:14,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:14,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:14,982 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:14,982 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:14,982 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:14,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:14,983 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:14,983 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:14,983 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/A is initiating minor compaction (all files) 2023-04-12 13:54:14,984 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/B is initiating minor compaction (all files) 2023-04-12 13:54:14,984 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/A in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,984 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/B in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:14,984 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/ab7a8bf15b0d4128a5b462726518b571, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ef84ae724c44baf9cc39a53ebf5b528, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c245e7b52b564d26bc74be17bab2c2c8] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=34.8 K 2023-04-12 13:54:14,984 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2c62f366ca204f88a2dcb507b2630439, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/46d8e43d4bb34293aaf80bdec48b3fb3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f4a521149fd741208a0400c6b33ab3cf] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=82.2 K 2023-04-12 13:54:14,984 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ab7a8bf15b0d4128a5b462726518b571, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681307651945 2023-04-12 13:54:14,984 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2c62f366ca204f88a2dcb507b2630439, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681307651945 2023-04-12 13:54:14,985 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7ef84ae724c44baf9cc39a53ebf5b528, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681307651981 2023-04-12 13:54:14,985 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 46d8e43d4bb34293aaf80bdec48b3fb3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681307651981 2023-04-12 13:54:14,985 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c245e7b52b564d26bc74be17bab2c2c8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307654117 2023-04-12 13:54:14,985 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f4a521149fd741208a0400c6b33ab3cf, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307654117 2023-04-12 13:54:14,995 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#B#compaction#120 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:15,010 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#A#compaction#121 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:15,014 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/7d12f59ef2af4b62b78c9e884f24212a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7d12f59ef2af4b62b78c9e884f24212a 2023-04-12 13:54:15,020 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/B of 6b28deeffcdc135c24d47af5464730df into 7d12f59ef2af4b62b78c9e884f24212a(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:15,021 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:15,021 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/B, priority=13, startTime=1681307654982; duration=0sec 2023-04-12 13:54:15,021 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:15,021 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:15,024 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:15,024 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/C is initiating minor compaction (all files) 2023-04-12 13:54:15,024 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/C in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:15,024 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a8d412e0f2a64ec784292fa2263289d7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67a8977a98bc4588ba389ea40daa1c67, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/90c839c7cbda4e1aaba7639df17b45fe] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=34.8 K 2023-04-12 13:54:15,024 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a8d412e0f2a64ec784292fa2263289d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681307651945 2023-04-12 13:54:15,025 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 67a8977a98bc4588ba389ea40daa1c67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681307651981 2023-04-12 13:54:15,025 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 90c839c7cbda4e1aaba7639df17b45fe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307654117 2023-04-12 13:54:15,036 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:15,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:15,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:15,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:15,048 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:15,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:15,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:15,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:15,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:15,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:15,050 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:15,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:15,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:15,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:15,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:15,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:15,056 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#C#compaction#122 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:15,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:15,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:15,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:15,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:15,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:15,067 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:15,067 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:15,067 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:15,067 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:15,067 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:54:15,068 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:15,068 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:15,068 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:15,068 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:15,068 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:15,068 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:15,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:15,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:15,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307715085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307715085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307715086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307715086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307715086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,106 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b557be98db4f4e74a2ad1bf2b744fbc0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b557be98db4f4e74a2ad1bf2b744fbc0 2023-04-12 13:54:15,133 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/a551f3514db546639650d4b8093827fa 2023-04-12 13:54:15,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307715191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307715191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307715191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307715191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307715192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307715392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307715394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307715394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307715394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307715396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,482 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/4602bf5f67794e5493e2a3e24d8d4005 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/4602bf5f67794e5493e2a3e24d8d4005 2023-04-12 13:54:15,488 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/C of 6b28deeffcdc135c24d47af5464730df into 4602bf5f67794e5493e2a3e24d8d4005(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:15,489 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:15,489 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/C, priority=13, startTime=1681307654983; duration=0sec 2023-04-12 13:54:15,489 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:15,546 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/dbcf2a03da6a4a89a9148afc76a9dd74 2023-04-12 13:54:15,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307715696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307715698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307715698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307715699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:15,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307715700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,866 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/75ac9fcc41bc445180fc99aa90710b94 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/75ac9fcc41bc445180fc99aa90710b94 2023-04-12 13:54:15,876 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/A of 6b28deeffcdc135c24d47af5464730df into 75ac9fcc41bc445180fc99aa90710b94(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:15,877 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:15,877 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/A, priority=13, startTime=1681307654982; duration=0sec 2023-04-12 13:54:15,877 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:15,970 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/cc74dc428ac3493ebce8bb57fca33e8d 2023-04-12 13:54:15,976 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/a551f3514db546639650d4b8093827fa as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a551f3514db546639650d4b8093827fa 2023-04-12 13:54:15,981 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a551f3514db546639650d4b8093827fa, entries=150, sequenceid=69, filesize=25.1 K 2023-04-12 13:54:15,982 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/dbcf2a03da6a4a89a9148afc76a9dd74 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/dbcf2a03da6a4a89a9148afc76a9dd74 2023-04-12 13:54:15,991 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/dbcf2a03da6a4a89a9148afc76a9dd74, entries=150, sequenceid=69, filesize=11.6 K 2023-04-12 13:54:15,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/cc74dc428ac3493ebce8bb57fca33e8d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/cc74dc428ac3493ebce8bb57fca33e8d 2023-04-12 13:54:15,997 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/cc74dc428ac3493ebce8bb57fca33e8d, entries=150, sequenceid=69, filesize=11.6 K 2023-04-12 13:54:15,998 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6b28deeffcdc135c24d47af5464730df in 931ms, sequenceid=69, compaction requested=false 2023-04-12 13:54:15,998 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:15,998 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:15,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:15,999 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:16,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:16,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:16,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,005 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:16,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:16,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9a0a52[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:16,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:16,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:16,008 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:16,008 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,008 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:16,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:16,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:16,011 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:16,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:16,015 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:16,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:16,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:16,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:16,015 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:16,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,017 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:16,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:16,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:16,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,019 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:16,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:16,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:16,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:16,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,025 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:16,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:16,025 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:16,126 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:16,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:16,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:16,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:16,130 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,130 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:16,134 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:16,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:16,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:16,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:16,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:16,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:16,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:16,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:16,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:16,138 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:16,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:16,139 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:16,172 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304124762fd370a134d86bf66ea6b5e41b7e4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124762fd370a134d86bf66ea6b5e41b7e4 2023-04-12 13:54:16,186 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/66798c05116e420e81efdca53e4881c9 2023-04-12 13:54:16,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:16,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:16,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307716248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307716248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307716275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307716275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307716275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307716376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307716376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307716380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307716380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307716380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307716577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307716578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307716581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307716582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307716583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,599 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/e92af0ba32e84b5aab581839978ce4da 2023-04-12 13:54:16,610 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/3373698d88fd4037ac289fc2aab41423 2023-04-12 13:54:16,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/66798c05116e420e81efdca53e4881c9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/66798c05116e420e81efdca53e4881c9 2023-04-12 13:54:16,619 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/66798c05116e420e81efdca53e4881c9, entries=150, sequenceid=84, filesize=25.1 K 2023-04-12 13:54:16,623 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/e92af0ba32e84b5aab581839978ce4da as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e92af0ba32e84b5aab581839978ce4da 2023-04-12 13:54:16,628 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e92af0ba32e84b5aab581839978ce4da, entries=150, sequenceid=84, filesize=11.6 K 2023-04-12 13:54:16,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/3373698d88fd4037ac289fc2aab41423 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/3373698d88fd4037ac289fc2aab41423 2023-04-12 13:54:16,636 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/3373698d88fd4037ac289fc2aab41423, entries=150, sequenceid=84, filesize=11.6 K 2023-04-12 13:54:16,638 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6b28deeffcdc135c24d47af5464730df in 500ms, sequenceid=84, compaction requested=true 2023-04-12 13:54:16,638 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:16,638 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:16,639 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:16,640 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:16,642 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:16,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:16,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:16,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:16,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:16,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34de57e3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:16,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:16,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,646 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:16,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:16,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:16,646 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,646 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:16,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,647 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:16,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:16,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:16,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:16,650 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:16,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:16,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:16,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:16,650 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:16,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:16,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:16,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:16,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:16,652 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:16,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:16,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,655 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:16,655 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:16,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:16,655 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,655 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,655 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:16,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:16,656 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:16,656 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:16,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:16,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:16,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,656 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:16,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:16,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:16,756 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:16,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:16,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:16,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:16,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:16,760 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:16,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:16,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:16,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:16,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,764 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:16,764 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:16,778 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,778 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:16,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:16,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:16,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:16,779 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:16,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:16,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:16,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:16,780 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:16,780 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:16,780 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:16,780 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:54:16,780 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:16,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:16,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:16,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:16,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:16,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:16,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:16,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:16,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307716889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307716890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307716892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307716893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307716893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307716993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307716994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307716996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307716997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:16,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:16,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307716997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,200 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304123807c9f8604c40f18cfe4997e8b25677 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123807c9f8604c40f18cfe4997e8b25677 2023-04-12 13:54:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307717197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307717197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307717201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307717203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307717203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307717502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307717504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307717504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307717504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:17,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307717505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,607 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/79353a99c5ee4f6eb0cb5f186c7f4a5f 2023-04-12 13:54:17,629 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3 2023-04-12 13:54:17,649 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/42cb16d50c034930adf8e3cd38beff70 2023-04-12 13:54:17,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/79353a99c5ee4f6eb0cb5f186c7f4a5f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/79353a99c5ee4f6eb0cb5f186c7f4a5f 2023-04-12 13:54:17,661 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/79353a99c5ee4f6eb0cb5f186c7f4a5f, entries=150, sequenceid=103, filesize=25.1 K 2023-04-12 13:54:17,661 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3 2023-04-12 13:54:17,667 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3, entries=150, sequenceid=103, filesize=11.6 K 2023-04-12 13:54:17,667 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/42cb16d50c034930adf8e3cd38beff70 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/42cb16d50c034930adf8e3cd38beff70 2023-04-12 13:54:17,672 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/42cb16d50c034930adf8e3cd38beff70, entries=150, sequenceid=103, filesize=11.6 K 2023-04-12 13:54:17,673 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 6b28deeffcdc135c24d47af5464730df in 893ms, sequenceid=103, compaction requested=true 2023-04-12 13:54:17,673 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:17,673 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:17,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:17,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:17,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:17,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:17,674 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,674 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:17,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:17,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:17,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:17,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6a7b00[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:17,676 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:17,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,677 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:17,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:17,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:17,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:17,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:17,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:17,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:17,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:17,682 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:17,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:17,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:17,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:17,682 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:17,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:17,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:17,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:17,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:17,686 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:17,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:17,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:17,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:17,689 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:17,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,689 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:17,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:17,789 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:17,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:17,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:17,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:17,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:17,793 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:17,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:17,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:17,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:17,794 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,794 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:17,795 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:17,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:17,796 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:17,796 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:17,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:17,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:17,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:17,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:17,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:17,798 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:17,798 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:17,798 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:17,798 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:17,799 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:17,818 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412c157233e36de46d086f27faf0d1ef2ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c157233e36de46d086f27faf0d1ef2ab 2023-04-12 13:54:18,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:18,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:18,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307718016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307718017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307718018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307718023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307718023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307718124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307718124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307718125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307718127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307718127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,224 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/3d0eb6ace92a413581b8a7e108315be4 2023-04-12 13:54:18,242 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/15f7b0e807c9456fbd118bf273758f5b 2023-04-12 13:54:18,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307718326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307718326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307718326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307718329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307718330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307718629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307718630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307718630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307718633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:18,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307718633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,656 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/70390f1699ff400e9c8556b15dc9ce54 2023-04-12 13:54:18,661 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/3d0eb6ace92a413581b8a7e108315be4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/3d0eb6ace92a413581b8a7e108315be4 2023-04-12 13:54:18,666 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/3d0eb6ace92a413581b8a7e108315be4, entries=150, sequenceid=117, filesize=25.1 K 2023-04-12 13:54:18,667 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/15f7b0e807c9456fbd118bf273758f5b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/15f7b0e807c9456fbd118bf273758f5b 2023-04-12 13:54:18,671 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/15f7b0e807c9456fbd118bf273758f5b, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:54:18,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/70390f1699ff400e9c8556b15dc9ce54 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/70390f1699ff400e9c8556b15dc9ce54 2023-04-12 13:54:18,676 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/70390f1699ff400e9c8556b15dc9ce54, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:54:18,677 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6b28deeffcdc135c24d47af5464730df in 879ms, sequenceid=117, compaction requested=true 2023-04-12 13:54:18,677 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:18,677 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:18,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:18,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:18,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:18,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1128fbb9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:18,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:18,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,683 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:18,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:18,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:18,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:18,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:18,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:18,687 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:18,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:18,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:18,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:18,687 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:18,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:18,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:18,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:18,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:18,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:18,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:18,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:18,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:18,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:18,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:18,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:18,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:18,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:18,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,693 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:18,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:18,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:18,693 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:18,793 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:18,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:18,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:18,798 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:18,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:18,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:18,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:18,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:18,800 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:18,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:18,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:18,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:18,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:18,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:18,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:18,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:18,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:18,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:18,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:18,803 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:54:18,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:18,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:18,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:18,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:18,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:18,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:19,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:19,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:19,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307719142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307719146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307719147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307719148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307719149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,218 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304122fd09a5c47754b8e9687b8a8295eeaf9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122fd09a5c47754b8e9687b8a8295eeaf9 2023-04-12 13:54:19,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307719249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307719249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307719250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307719251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307719251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307719452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307719452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307719453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307719453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307719454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,624 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/06b7e095c32a41e59753a80bbdb38f00 2023-04-12 13:54:19,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307719754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307719755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307719755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307719757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:19,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:19,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307719759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,036 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/151936e2e3194e7eba53639de080950a 2023-04-12 13:54:20,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:20,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307720260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:20,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307720260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:20,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307720261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:20,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307720263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307720265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,449 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/8ba4f4f61c9b48a79e54a38a3274f055 2023-04-12 13:54:20,455 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/06b7e095c32a41e59753a80bbdb38f00 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/06b7e095c32a41e59753a80bbdb38f00 2023-04-12 13:54:20,463 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/06b7e095c32a41e59753a80bbdb38f00, entries=150, sequenceid=135, filesize=25.3 K 2023-04-12 13:54:20,464 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/151936e2e3194e7eba53639de080950a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/151936e2e3194e7eba53639de080950a 2023-04-12 13:54:20,469 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/151936e2e3194e7eba53639de080950a, entries=150, sequenceid=135, filesize=11.8 K 2023-04-12 13:54:20,469 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/8ba4f4f61c9b48a79e54a38a3274f055 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/8ba4f4f61c9b48a79e54a38a3274f055 2023-04-12 13:54:20,475 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/8ba4f4f61c9b48a79e54a38a3274f055, entries=150, sequenceid=135, filesize=11.8 K 2023-04-12 13:54:20,476 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 6b28deeffcdc135c24d47af5464730df in 1673ms, sequenceid=135, compaction requested=true 2023-04-12 13:54:20,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:20,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:20,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:20,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:20,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:20,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,482 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1dbb54[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:20,484 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:20,484 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,485 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:20,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:20,487 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:20,490 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:20,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:20,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:20,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:20,490 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:20,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:20,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:20,494 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,497 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:20,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:20,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:20,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:20,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,500 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:20,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:20,500 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:20,601 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:20,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:20,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:20,606 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:20,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:20,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:20,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:20,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:20,608 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,608 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:20,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:20,608 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:20,609 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:20,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:20,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:20,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:20,610 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:20,610 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:20,610 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:20,610 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:54:20,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:20,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:20,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:20,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:20,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:20,611 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:21,043 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304122c9ec9bac4a2470a904a7081a9c1c007 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122c9ec9bac4a2470a904a7081a9c1c007 2023-04-12 13:54:21,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:21,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:21,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307721276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307721277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307721280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307721281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307721281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307721382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307721382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307721385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307721386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307721386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,449 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/92975ac385404ba580e51b7a806af7af 2023-04-12 13:54:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307721588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307721589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307721589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307721589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307721590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,862 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/7ffe2e3d38284cc4bc9964c6848c3d63 2023-04-12 13:54:21,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307721893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307721893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307721893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307721895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:21,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:21,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307721895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,274 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/187806e62e4e4089b2fda2fc8d5dd5cf 2023-04-12 13:54:22,279 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/92975ac385404ba580e51b7a806af7af as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/92975ac385404ba580e51b7a806af7af 2023-04-12 13:54:22,284 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/92975ac385404ba580e51b7a806af7af, entries=150, sequenceid=150, filesize=25.3 K 2023-04-12 13:54:22,285 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/7ffe2e3d38284cc4bc9964c6848c3d63 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ffe2e3d38284cc4bc9964c6848c3d63 2023-04-12 13:54:22,289 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ffe2e3d38284cc4bc9964c6848c3d63, entries=150, sequenceid=150, filesize=11.8 K 2023-04-12 13:54:22,290 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/187806e62e4e4089b2fda2fc8d5dd5cf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/187806e62e4e4089b2fda2fc8d5dd5cf 2023-04-12 13:54:22,295 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/187806e62e4e4089b2fda2fc8d5dd5cf, entries=150, sequenceid=150, filesize=11.8 K 2023-04-12 13:54:22,296 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6b28deeffcdc135c24d47af5464730df in 1686ms, sequenceid=150, compaction requested=true 2023-04-12 13:54:22,296 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:22,296 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:22,298 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,298 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:22,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3a987b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:22,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:22,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:22,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:22,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:22,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:22,307 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:22,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:22,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:22,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:22,307 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:22,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:22,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,314 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,314 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:22,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:54:22,398 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:22,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:22,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:22,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:22,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:22,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:22,414 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:22,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:22,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307722412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:22,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:22,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:22,418 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:22,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:22,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:22,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:22,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:22,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:22,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:22,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:22,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:22,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:22,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307722415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:22,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e05447[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,427 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:22,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:22,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307722416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307722416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307722417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,429 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:22,431 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:22,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:22,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:22,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:22,431 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:22,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:22,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,437 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:22,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:22,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,438 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:22,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307722518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307722524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307722529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307722529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307722529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,538 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:22,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:22,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:22,541 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:22,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:22,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:22,545 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,545 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:22,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:22,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:22,547 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:22,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c39a91[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:22,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:22,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,551 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:22,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:22,556 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:22,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:22,558 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:22,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:22,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:22,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:22,559 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:22,560 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,560 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,560 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:22,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:22,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,564 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:22,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,564 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:22,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,665 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:22,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:22,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:22,668 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:22,668 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,668 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:22,668 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:22,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,670 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,671 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:22,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:22,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:22,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:22,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:22,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:22,672 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:22,675 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:22,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15855446[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:22,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:22,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:22,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:22,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:22,682 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:22,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:22,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:22,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:22,682 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:22,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:22,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:22,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,689 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:22,689 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:22,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307722720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307722727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307722732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307722733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:22,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307722733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,789 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:22,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:22,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:22,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:22,793 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:22,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:22,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:22,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,795 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,795 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,795 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,795 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:22,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:22,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:22,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:22,796 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,797 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:22,797 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:22,797 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:22,799 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,800 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:22,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b82efcd[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:22,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:22,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,802 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:22,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:22,803 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,803 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:22,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:22,808 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:22,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:22,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:22,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:22,808 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:22,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,812 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,814 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d10837337f2640d48211c43a377625db to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d10837337f2640d48211c43a377625db 2023-04-12 13:54:22,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:22,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:22,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:22,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,816 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,816 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:22,816 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:22,916 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:22,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:22,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:22,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:22,920 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:22,920 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,920 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:22,920 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:22,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:22,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:22,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:22,924 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,924 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:22,924 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:22,924 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:22,927 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:22,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b80dc96[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:22,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:22,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,931 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:22,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:22,934 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:22,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:22,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:22,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:22,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:22,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:22,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:22,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:22,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:22,945 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:22,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:22,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:22,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:22,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:22,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,950 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:22,950 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:22,950 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,950 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:22,950 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:22,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:22,950 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:22,950 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:22,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307723023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307723033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307723037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307723037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307723039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,050 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,053 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,058 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231fcc65[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,062 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,062 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,063 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,066 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,066 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,067 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,067 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,069 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:23,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,072 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,072 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,172 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,176 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,179 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,181 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,181 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,181 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,181 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,183 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c078b24[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,190 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,190 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,197 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,197 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,221 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/835bbe12afb245bc803c4fb3facdc758 2023-04-12 13:54:23,297 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,300 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,302 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,306 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,306 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6007f3ee[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,314 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,314 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,317 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,322 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,329 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,329 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,330 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,430 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,433 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,435 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,437 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,438 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,438 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,438 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcee714[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,443 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,443 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,443 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,443 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,453 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,453 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,454 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,454 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,454 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,456 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,457 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,460 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:23,460 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,460 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307723526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307723539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307723539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307723540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307723546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,560 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,564 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,565 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,567 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,568 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,568 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,568 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106f762c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,573 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,577 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,577 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,581 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,585 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,585 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c654c02ebb784de49e838f0b562743de 2023-04-12 13:54:23,685 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,689 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,690 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,693 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,693 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,693 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,693 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301191ae[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,698 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,699 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,700 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,700 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,702 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,705 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,705 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,705 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,705 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,705 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,715 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,715 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,715 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,715 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,715 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,715 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,715 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,715 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:23,815 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,819 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,821 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,821 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,823 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,823 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,823 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f569cc1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,835 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,835 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,853 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,853 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,857 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:23,864 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,864 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,864 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,964 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:23,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:23,968 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:23,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:23,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:23,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,970 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:23,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:23,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:23,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:23,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559a2078[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:23,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:23,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:23,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:23,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:23,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,979 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:23,982 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:23,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:23,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:23,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:23,982 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:23,983 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,983 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,983 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,984 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:23,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,989 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:23,989 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:23,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:23,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:23,989 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:23,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:23,989 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:23,990 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:23,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:23,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:24,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/2e65cbf4ff224c33a05f8d4098199732 2023-04-12 13:54:24,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/835bbe12afb245bc803c4fb3facdc758 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/835bbe12afb245bc803c4fb3facdc758 2023-04-12 13:54:24,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/835bbe12afb245bc803c4fb3facdc758, entries=200, sequenceid=168, filesize=32.1 K 2023-04-12 13:54:24,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c654c02ebb784de49e838f0b562743de as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c654c02ebb784de49e838f0b562743de 2023-04-12 13:54:24,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c654c02ebb784de49e838f0b562743de, entries=150, sequenceid=168, filesize=11.8 K 2023-04-12 13:54:24,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/2e65cbf4ff224c33a05f8d4098199732 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/2e65cbf4ff224c33a05f8d4098199732 2023-04-12 13:54:24,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/2e65cbf4ff224c33a05f8d4098199732, entries=150, sequenceid=168, filesize=11.8 K 2023-04-12 13:54:24,069 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6b28deeffcdc135c24d47af5464730df in 1671ms, sequenceid=168, compaction requested=true 2023-04-12 13:54:24,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:24,069 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-12 13:54:24,070 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:24,070 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:24,070 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:24,070 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-12 13:54:24,072 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 213511 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-12 13:54:24,073 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/A is initiating minor compaction (all files) 2023-04-12 13:54:24,073 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/A in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:24,073 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/75ac9fcc41bc445180fc99aa90710b94, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a551f3514db546639650d4b8093827fa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/66798c05116e420e81efdca53e4881c9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/79353a99c5ee4f6eb0cb5f186c7f4a5f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/3d0eb6ace92a413581b8a7e108315be4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/06b7e095c32a41e59753a80bbdb38f00, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/92975ac385404ba580e51b7a806af7af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/835bbe12afb245bc803c4fb3facdc758] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=208.5 K 2023-04-12 13:54:24,073 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95641 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-12 13:54:24,073 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/B is initiating minor compaction (all files) 2023-04-12 13:54:24,073 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/B in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:24,074 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7d12f59ef2af4b62b78c9e884f24212a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/dbcf2a03da6a4a89a9148afc76a9dd74, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e92af0ba32e84b5aab581839978ce4da, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/15f7b0e807c9456fbd118bf273758f5b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/151936e2e3194e7eba53639de080950a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ffe2e3d38284cc4bc9964c6848c3d63, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c654c02ebb784de49e838f0b562743de] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=93.4 K 2023-04-12 13:54:24,075 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 75ac9fcc41bc445180fc99aa90710b94, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307654117 2023-04-12 13:54:24,075 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7d12f59ef2af4b62b78c9e884f24212a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307654117 2023-04-12 13:54:24,075 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a551f3514db546639650d4b8093827fa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681307654455 2023-04-12 13:54:24,075 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting dbcf2a03da6a4a89a9148afc76a9dd74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681307654455 2023-04-12 13:54:24,076 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e92af0ba32e84b5aab581839978ce4da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681307655083 2023-04-12 13:54:24,076 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 66798c05116e420e81efdca53e4881c9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681307655083 2023-04-12 13:54:24,077 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a9e682d0a0ee4bf0b6754d5c20ed4fc3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681307656248 2023-04-12 13:54:24,077 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 79353a99c5ee4f6eb0cb5f186c7f4a5f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681307656248 2023-04-12 13:54:24,079 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 15f7b0e807c9456fbd118bf273758f5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307656889 2023-04-12 13:54:24,079 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3d0eb6ace92a413581b8a7e108315be4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307656889 2023-04-12 13:54:24,079 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 151936e2e3194e7eba53639de080950a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681307658015 2023-04-12 13:54:24,080 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 06b7e095c32a41e59753a80bbdb38f00, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681307658015 2023-04-12 13:54:24,080 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7ffe2e3d38284cc4bc9964c6848c3d63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681307659141 2023-04-12 13:54:24,080 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 92975ac385404ba580e51b7a806af7af, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681307659141 2023-04-12 13:54:24,081 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c654c02ebb784de49e838f0b562743de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681307661279 2023-04-12 13:54:24,081 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 835bbe12afb245bc803c4fb3facdc758, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681307661275 2023-04-12 13:54:24,090 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:24,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:24,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:24,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:24,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:24,095 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:24,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:24,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:24,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:24,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:24,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:24,098 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,098 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:24,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:24,098 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,098 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:24,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:24,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:24,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:24,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:24,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:24,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:24,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:24,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:24,101 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:24,101 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:54:24,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:24,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:24,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:24,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:24,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:24,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:24,111 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#B#compaction#145 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:24,135 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#A#compaction#144 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:24,144 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/e1f26133683c4e5e8020f9911a696417 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e1f26133683c4e5e8020f9911a696417 2023-04-12 13:54:24,151 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/B of 6b28deeffcdc135c24d47af5464730df into e1f26133683c4e5e8020f9911a696417(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:24,151 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:24,151 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/B, priority=8, startTime=1681307664070; duration=0sec 2023-04-12 13:54:24,151 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:24,152 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-12 13:54:24,155 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95641 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-12 13:54:24,155 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/C is initiating minor compaction (all files) 2023-04-12 13:54:24,155 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/C in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:24,155 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/4602bf5f67794e5493e2a3e24d8d4005, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/cc74dc428ac3493ebce8bb57fca33e8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/3373698d88fd4037ac289fc2aab41423, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/42cb16d50c034930adf8e3cd38beff70, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/70390f1699ff400e9c8556b15dc9ce54, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/8ba4f4f61c9b48a79e54a38a3274f055, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/187806e62e4e4089b2fda2fc8d5dd5cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/2e65cbf4ff224c33a05f8d4098199732] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=93.4 K 2023-04-12 13:54:24,155 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4602bf5f67794e5493e2a3e24d8d4005, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307654117 2023-04-12 13:54:24,156 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting cc74dc428ac3493ebce8bb57fca33e8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681307654455 2023-04-12 13:54:24,156 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3373698d88fd4037ac289fc2aab41423, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681307655083 2023-04-12 13:54:24,156 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 42cb16d50c034930adf8e3cd38beff70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681307656248 2023-04-12 13:54:24,157 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 70390f1699ff400e9c8556b15dc9ce54, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307656889 2023-04-12 13:54:24,157 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8ba4f4f61c9b48a79e54a38a3274f055, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681307658015 2023-04-12 13:54:24,157 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 187806e62e4e4089b2fda2fc8d5dd5cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681307659141 2023-04-12 13:54:24,158 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2e65cbf4ff224c33a05f8d4098199732, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681307661279 2023-04-12 13:54:24,169 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#C#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:24,182 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/a93e54c972cd4ebaa6c3f8aa90f377f7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a93e54c972cd4ebaa6c3f8aa90f377f7 2023-04-12 13:54:24,188 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/C of 6b28deeffcdc135c24d47af5464730df into a93e54c972cd4ebaa6c3f8aa90f377f7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:24,188 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:24,188 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/C, priority=8, startTime=1681307664070; duration=0sec 2023-04-12 13:54:24,188 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:24,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:24,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:24,544 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041238bf7a5b28b645aab59e997368e0d670 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041238bf7a5b28b645aab59e997368e0d670 2023-04-12 13:54:24,558 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/d47133d15ac1406d99c7c9f6551d7f84 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d47133d15ac1406d99c7c9f6551d7f84 2023-04-12 13:54:24,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307724556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307724557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307724557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307724558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307724558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,563 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/A of 6b28deeffcdc135c24d47af5464730df into d47133d15ac1406d99c7c9f6551d7f84(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:24,563 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:24,563 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/A, priority=8, startTime=1681307664069; duration=0sec 2023-04-12 13:54:24,563 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:24,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307724659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307724660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307724660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307724661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307724662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307724861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307724862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307724862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307724863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:24,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307724864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:24,951 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/e3846261dbfc48759e1937e7d236213e 2023-04-12 13:54:25,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307725163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307725164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307725165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307725165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307725167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,365 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/833310f730674389a8b5298f3dac39d6 2023-04-12 13:54:25,385 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/67b48a55ab3e4ab8b9471582b72d8c6d 2023-04-12 13:54:25,390 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/e3846261dbfc48759e1937e7d236213e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/e3846261dbfc48759e1937e7d236213e 2023-04-12 13:54:25,395 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/e3846261dbfc48759e1937e7d236213e, entries=150, sequenceid=182, filesize=25.3 K 2023-04-12 13:54:25,396 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/833310f730674389a8b5298f3dac39d6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/833310f730674389a8b5298f3dac39d6 2023-04-12 13:54:25,400 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/833310f730674389a8b5298f3dac39d6, entries=150, sequenceid=182, filesize=11.8 K 2023-04-12 13:54:25,401 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/67b48a55ab3e4ab8b9471582b72d8c6d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67b48a55ab3e4ab8b9471582b72d8c6d 2023-04-12 13:54:25,405 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67b48a55ab3e4ab8b9471582b72d8c6d, entries=150, sequenceid=182, filesize=11.8 K 2023-04-12 13:54:25,406 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6b28deeffcdc135c24d47af5464730df in 1305ms, sequenceid=182, compaction requested=false 2023-04-12 13:54:25,406 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:25,406 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:25,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:25,407 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:25,408 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:25,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:25,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eafe2f1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:25,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:25,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:25,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:25,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:25,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:25,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:25,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:25,413 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:25,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:25,422 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:25,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:25,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:25,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:25,422 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:25,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:25,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:25,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,428 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:25,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:25,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:25,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:25,430 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:25,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,430 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:25,530 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:25,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:25,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:25,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:25,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:25,534 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:25,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:25,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:25,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:25,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:25,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:25,536 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:25,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:25,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:25,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:25,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:25,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:25,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:25,538 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:25,538 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:25,538 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:54:25,538 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:25,538 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:25,538 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:25,538 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:25,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:25,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:25,561 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b0593c8a25844a8c9edbf910c2514f9c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b0593c8a25844a8c9edbf910c2514f9c 2023-04-12 13:54:25,568 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/2a1c2715fab44cfbb82236b6034afb6c 2023-04-12 13:54:25,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:25,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:25,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307725675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307725675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307725676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307725676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307725677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307725778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307725778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307725779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36108 deadline: 1681307725779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307725779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,983 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/45f05efa74594a5c847fa8bfd6b5266c 2023-04-12 13:54:25,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307725982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307725982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36118 deadline: 1681307725983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307725983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307725983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307726286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36160 deadline: 1681307726286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36176 deadline: 1681307726286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36132 deadline: 1681307726287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307726287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,396 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/210092fc6c88419b85739dea6cb0cc19 2023-04-12 13:54:26,401 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/2a1c2715fab44cfbb82236b6034afb6c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2a1c2715fab44cfbb82236b6034afb6c 2023-04-12 13:54:26,405 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2a1c2715fab44cfbb82236b6034afb6c, entries=150, sequenceid=203, filesize=25.3 K 2023-04-12 13:54:26,406 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/45f05efa74594a5c847fa8bfd6b5266c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/45f05efa74594a5c847fa8bfd6b5266c 2023-04-12 13:54:26,411 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/45f05efa74594a5c847fa8bfd6b5266c, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 13:54:26,412 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/210092fc6c88419b85739dea6cb0cc19 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/210092fc6c88419b85739dea6cb0cc19 2023-04-12 13:54:26,416 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/210092fc6c88419b85739dea6cb0cc19, entries=150, sequenceid=203, filesize=11.8 K 2023-04-12 13:54:26,417 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 6b28deeffcdc135c24d47af5464730df in 879ms, sequenceid=203, compaction requested=true 2023-04-12 13:54:26,417 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:26,417 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:26,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:26,422 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f886b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:26,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:26,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:26,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:26,427 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:26,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:26,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:26,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:26,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:26,430 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:26,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:26,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:26,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:26,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:26,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:26,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:26,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:26,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:26,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:26,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:26,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:26,438 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:26,438 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:26,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:26,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:26,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:26,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:26,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,539 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:26,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:26,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:26,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:26,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:26,542 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:26,542 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:26,542 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:26,542 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:26,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:26,544 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:26,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:26,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:26,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:26,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:26,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:26,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:26,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:26,546 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:26,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:26,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:26,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307726800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307726801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307726802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307726803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307726805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307726905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307726905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307726905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307726905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:26,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307726909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:26,964 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e0856cf8913244238a2154a2f15d4109 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e0856cf8913244238a2154a2f15d4109 2023-04-12 13:54:27,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307727108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307727108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307727109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307727109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307727113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,370 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/192aa925683148d49442ce09d49b78ee 2023-04-12 13:54:27,383 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c3462217efe54094bc7017f164ef357b 2023-04-12 13:54:27,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307727411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307727413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307727414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307727414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307727418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,795 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/e51db76332c94a44ac279f34bc1e6710 2023-04-12 13:54:27,800 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/192aa925683148d49442ce09d49b78ee as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/192aa925683148d49442ce09d49b78ee 2023-04-12 13:54:27,804 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/192aa925683148d49442ce09d49b78ee, entries=150, sequenceid=218, filesize=25.3 K 2023-04-12 13:54:27,805 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c3462217efe54094bc7017f164ef357b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c3462217efe54094bc7017f164ef357b 2023-04-12 13:54:27,810 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c3462217efe54094bc7017f164ef357b, entries=150, sequenceid=218, filesize=11.8 K 2023-04-12 13:54:27,811 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/e51db76332c94a44ac279f34bc1e6710 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/e51db76332c94a44ac279f34bc1e6710 2023-04-12 13:54:27,815 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/e51db76332c94a44ac279f34bc1e6710, entries=150, sequenceid=218, filesize=11.8 K 2023-04-12 13:54:27,816 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6b28deeffcdc135c24d47af5464730df in 1270ms, sequenceid=218, compaction requested=true 2023-04-12 13:54:27,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:27,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:27,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:27,818 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:27,818 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:27,818 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,818 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:27,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:27,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5964bc7f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:27,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:27,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:27,820 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:27,821 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,821 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:27,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:27,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:27,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:27,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:27,825 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:27,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:27,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:27,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:27,825 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:27,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:27,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:27,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:27,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:27,831 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:27,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:27,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:27,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:27,831 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:27,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,832 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:27,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:27,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:54:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:27,921 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:27,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:27,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:27,932 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:27,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307727926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307727930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307727930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307727931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:27,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307727932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:27,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:27,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:27,936 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:27,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:27,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:27,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:27,936 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412dad6ba252b194c93bde768291c5ed206 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412dad6ba252b194c93bde768291c5ed206 2023-04-12 13:54:27,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,941 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,941 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:27,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:27,941 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,941 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:27,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:27,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:27,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:27,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:27,951 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:27,951 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,953 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,953 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d79cecb[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:27,955 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:27,955 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,957 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,957 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:27,957 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:27,957 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:27,957 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,957 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:27,959 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:27,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:27,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:27,961 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:27,961 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:27,961 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:27,961 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:27,961 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:27,963 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,963 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,963 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:27,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:27,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:27,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:27,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:27,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:27,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:27,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:27,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:27,968 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:27,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:27,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:27,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:27,968 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:27,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307728034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307728034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307728034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307728037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307728037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,068 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,071 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,077 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,077 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,077 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,077 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,078 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c0007ff[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,080 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,080 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,085 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,085 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,085 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,087 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,087 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,087 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,090 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,090 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,094 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,096 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,096 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,096 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:28,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,197 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,200 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,202 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,202 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,202 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,204 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,204 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,204 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,204 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b298c4[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,208 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,208 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,209 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,209 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,209 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,209 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,209 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,209 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,211 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,211 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,213 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,214 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,215 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,215 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,215 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,216 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,220 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,220 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,220 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,220 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,220 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,220 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,220 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307728239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307728239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307728240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307728240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307728240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,320 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,324 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,325 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,325 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,326 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,326 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,329 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4947a9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,332 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,333 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,333 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,335 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,335 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,337 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,338 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,338 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,338 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,338 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,339 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,339 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,339 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,340 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,341 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,343 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,343 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,343 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,343 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,343 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:28,343 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,343 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,344 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,344 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,344 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,344 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,344 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,353 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/6507df9b08c54b8f9e1d17aa3acd2680 2023-04-12 13:54:28,444 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,448 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,450 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,450 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,450 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,452 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,452 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,452 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,452 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d55cd3d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,456 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,457 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,458 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,459 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,462 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,462 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,462 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,462 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,462 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,472 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,472 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,472 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,474 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:28,477 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,477 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,477 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,477 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,477 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,477 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307728541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307728542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307728542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307728544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:28,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307728544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,577 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,581 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,581 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,581 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,581 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,581 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,581 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,582 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,582 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,584 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,584 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,584 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,584 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,588 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11761c97[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,588 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,590 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,591 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,591 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,594 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,594 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,594 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,594 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,594 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,597 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,601 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:28,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,602 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,602 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,702 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,705 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,705 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,711 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,713 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,713 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,713 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,713 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,715 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6774c2c5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,717 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,717 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,719 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,721 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,721 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,724 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,724 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,724 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,725 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,731 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,734 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,734 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,734 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/e00066a2b167411982e00a35884227af 2023-04-12 13:54:28,834 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,838 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,838 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,838 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,838 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,838 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,838 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,839 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,841 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,841 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,841 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,841 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d4a58d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,846 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,850 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,850 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,852 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,852 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,852 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,852 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,861 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,861 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,861 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,961 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:28,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:28,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:28,964 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:28,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:28,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:28,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:28,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:28,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:28,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:28,970 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,970 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:28,970 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:28,970 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:28,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10cbbaef[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:28,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:28,974 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,975 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,975 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:28,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:28,980 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:28,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:28,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:28,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:28,980 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:28,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:28,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:28,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:28,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:28,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:28,986 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:28,986 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:28,986 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:28,987 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:29,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307729044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307729045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307729045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:29,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307729046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307729048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,086 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:29,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:29,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:29,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:29,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:29,090 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:29,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:29,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:29,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:29,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,090 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,091 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,091 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:29,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:29,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:29,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:29,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:29,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:29,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:29,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:29,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:29,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:29,094 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:29,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:29,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:29,098 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:29,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31788ec5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:29,098 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,099 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:29,099 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:29,099 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:29,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:29,101 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:29,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:29,103 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:29,103 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:29,103 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:29,103 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:29,103 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:29,104 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,104 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,104 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:29,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,105 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:29,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:29,106 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:29,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:29,109 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:29,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:29,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,109 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:29,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,109 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:29,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/796f7a714e444059b6b4e5efd8e31225 2023-04-12 13:54:29,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/6507df9b08c54b8f9e1d17aa3acd2680 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/6507df9b08c54b8f9e1d17aa3acd2680 2023-04-12 13:54:29,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/6507df9b08c54b8f9e1d17aa3acd2680, entries=200, sequenceid=237, filesize=32.1 K 2023-04-12 13:54:29,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/e00066a2b167411982e00a35884227af as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e00066a2b167411982e00a35884227af 2023-04-12 13:54:29,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e00066a2b167411982e00a35884227af, entries=150, sequenceid=237, filesize=11.8 K 2023-04-12 13:54:29,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/796f7a714e444059b6b4e5efd8e31225 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/796f7a714e444059b6b4e5efd8e31225 2023-04-12 13:54:29,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/796f7a714e444059b6b4e5efd8e31225, entries=150, sequenceid=237, filesize=11.8 K 2023-04-12 13:54:29,205 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6b28deeffcdc135c24d47af5464730df in 1285ms, sequenceid=237, compaction requested=true 2023-04-12 13:54:29,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:29,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:29,205 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:29,205 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:29,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:29,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:29,209 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:29,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:29,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:29,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:29,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:29,213 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:29,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:29,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:29,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:29,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,214 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:29,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:29,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:29,215 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:29,215 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:29,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:29,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:29,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:29,216 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:29,216 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:29,216 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:29,216 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:29,216 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:29,216 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:29,216 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:29,216 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:29,217 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136846 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:29,217 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/A is initiating minor compaction (all files) 2023-04-12 13:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:29,217 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/A in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:29,217 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:29,217 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d47133d15ac1406d99c7c9f6551d7f84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/e3846261dbfc48759e1937e7d236213e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2a1c2715fab44cfbb82236b6034afb6c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/192aa925683148d49442ce09d49b78ee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/6507df9b08c54b8f9e1d17aa3acd2680] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=133.6 K 2023-04-12 13:54:29,218 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60556 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:29,218 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/B is initiating minor compaction (all files) 2023-04-12 13:54:29,218 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/B in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:29,218 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e1f26133683c4e5e8020f9911a696417, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/833310f730674389a8b5298f3dac39d6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/45f05efa74594a5c847fa8bfd6b5266c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c3462217efe54094bc7017f164ef357b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e00066a2b167411982e00a35884227af] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=59.1 K 2023-04-12 13:54:29,218 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d47133d15ac1406d99c7c9f6551d7f84, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681307661279 2023-04-12 13:54:29,218 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e1f26133683c4e5e8020f9911a696417, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681307661279 2023-04-12 13:54:29,219 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e3846261dbfc48759e1937e7d236213e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307662410 2023-04-12 13:54:29,219 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 833310f730674389a8b5298f3dac39d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307662410 2023-04-12 13:54:29,220 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2a1c2715fab44cfbb82236b6034afb6c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681307664554 2023-04-12 13:54:29,220 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 45f05efa74594a5c847fa8bfd6b5266c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681307664554 2023-04-12 13:54:29,220 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 192aa925683148d49442ce09d49b78ee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681307665673 2023-04-12 13:54:29,221 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c3462217efe54094bc7017f164ef357b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681307665673 2023-04-12 13:54:29,221 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6507df9b08c54b8f9e1d17aa3acd2680, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681307666800 2023-04-12 13:54:29,222 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e00066a2b167411982e00a35884227af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681307666802 2023-04-12 13:54:29,238 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#B#compaction#161 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:29,250 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#A#compaction#160 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:29,258 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/4146d79ca02247c99fdbd80b52498ad2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/4146d79ca02247c99fdbd80b52498ad2 2023-04-12 13:54:29,264 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/B of 6b28deeffcdc135c24d47af5464730df into 4146d79ca02247c99fdbd80b52498ad2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:29,264 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:29,264 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/B, priority=11, startTime=1681307669205; duration=0sec 2023-04-12 13:54:29,264 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:29,264 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:29,267 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60556 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:29,267 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 6b28deeffcdc135c24d47af5464730df/C is initiating minor compaction (all files) 2023-04-12 13:54:29,267 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6b28deeffcdc135c24d47af5464730df/C in TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:29,267 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a93e54c972cd4ebaa6c3f8aa90f377f7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67b48a55ab3e4ab8b9471582b72d8c6d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/210092fc6c88419b85739dea6cb0cc19, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/e51db76332c94a44ac279f34bc1e6710, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/796f7a714e444059b6b4e5efd8e31225] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp, totalSize=59.1 K 2023-04-12 13:54:29,268 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a93e54c972cd4ebaa6c3f8aa90f377f7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681307661279 2023-04-12 13:54:29,268 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 67b48a55ab3e4ab8b9471582b72d8c6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307662410 2023-04-12 13:54:29,269 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 210092fc6c88419b85739dea6cb0cc19, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681307664554 2023-04-12 13:54:29,269 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e51db76332c94a44ac279f34bc1e6710, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681307665673 2023-04-12 13:54:29,269 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 796f7a714e444059b6b4e5efd8e31225, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1681307666802 2023-04-12 13:54:29,281 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 6b28deeffcdc135c24d47af5464730df#C#compaction#162 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:29,287 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/cb2396cba9114b42a69b054cda9e4db5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/cb2396cba9114b42a69b054cda9e4db5 2023-04-12 13:54:29,293 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/A of 6b28deeffcdc135c24d47af5464730df into cb2396cba9114b42a69b054cda9e4db5(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:29,293 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:29,293 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/A, priority=11, startTime=1681307669205; duration=0sec 2023-04-12 13:54:29,294 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:29,295 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/f7a92c9dbd154663a779dfbc11aae3df as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/f7a92c9dbd154663a779dfbc11aae3df 2023-04-12 13:54:29,300 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6b28deeffcdc135c24d47af5464730df/C of 6b28deeffcdc135c24d47af5464730df into f7a92c9dbd154663a779dfbc11aae3df(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:29,300 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:29,300 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df., storeName=6b28deeffcdc135c24d47af5464730df/C, priority=11, startTime=1681307669205; duration=0sec 2023-04-12 13:54:29,300 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:29,649 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127d8e39e1d8b14b53ae651d8ec71cb970 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127d8e39e1d8b14b53ae651d8ec71cb970 2023-04-12 13:54:30,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:30,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:30,056 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/a8ceb17e14de44158c43b6a2a1a9c3f7 2023-04-12 13:54:30,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307730063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307730063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307730063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307730065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307730066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307730167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307730167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307730167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307730169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307730171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307730369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307730371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307730371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307730371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307730374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,468 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/efbf86442d624787acbd0cb86e5bd889 2023-04-12 13:54:30,481 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/715cc48d6088425e9c1a6a0bd0d2d3c1 2023-04-12 13:54:30,486 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/a8ceb17e14de44158c43b6a2a1a9c3f7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a8ceb17e14de44158c43b6a2a1a9c3f7 2023-04-12 13:54:30,490 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a8ceb17e14de44158c43b6a2a1a9c3f7, entries=150, sequenceid=250, filesize=25.3 K 2023-04-12 13:54:30,491 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/efbf86442d624787acbd0cb86e5bd889 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/efbf86442d624787acbd0cb86e5bd889 2023-04-12 13:54:30,510 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/efbf86442d624787acbd0cb86e5bd889, entries=150, sequenceid=250, filesize=11.8 K 2023-04-12 13:54:30,512 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/715cc48d6088425e9c1a6a0bd0d2d3c1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/715cc48d6088425e9c1a6a0bd0d2d3c1 2023-04-12 13:54:30,518 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/715cc48d6088425e9c1a6a0bd0d2d3c1, entries=150, sequenceid=250, filesize=11.8 K 2023-04-12 13:54:30,518 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6b28deeffcdc135c24d47af5464730df in 1302ms, sequenceid=250, compaction requested=false 2023-04-12 13:54:30,519 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:30,519 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:30,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:30,522 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:30,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:30,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705b00c5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:30,524 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:30,524 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,525 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:30,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:30,528 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:30,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:30,531 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:30,531 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:30,531 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:30,531 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:30,531 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:30,533 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,533 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,533 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:30,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:30,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:30,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:30,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:30,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:30,540 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:30,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:30,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:30,541 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:30,541 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,541 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:30,541 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:30,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:30,641 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:30,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:30,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:30,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:30,644 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:30,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:30,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:30,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:30,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:30,646 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:30,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:30,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:30,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:30,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:30,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:30,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:30,648 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:30,648 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:30,648 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:30,649 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:30,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:30,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:30,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307730680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307730681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307730681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307730682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307730685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307730786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307730786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307730786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307730786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307730788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307730988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307730989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307730988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307730990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307730990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,066 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412f38593c7ced04e4bb5cfb2488a6eb1c0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f38593c7ced04e4bb5cfb2488a6eb1c0 2023-04-12 13:54:31,071 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/caa63b76c2cf4fe2941c594a46806789 2023-04-12 13:54:31,082 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/b9683664dd5d4ca49ff179d65d6548ce 2023-04-12 13:54:31,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307731292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307731293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307731294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307731294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307731294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,492 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/24d8ab71f01c4740a6331035f0e319eb 2023-04-12 13:54:31,497 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/caa63b76c2cf4fe2941c594a46806789 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/caa63b76c2cf4fe2941c594a46806789 2023-04-12 13:54:31,501 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/caa63b76c2cf4fe2941c594a46806789, entries=150, sequenceid=272, filesize=25.4 K 2023-04-12 13:54:31,502 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/b9683664dd5d4ca49ff179d65d6548ce as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/b9683664dd5d4ca49ff179d65d6548ce 2023-04-12 13:54:31,507 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/b9683664dd5d4ca49ff179d65d6548ce, entries=150, sequenceid=272, filesize=11.9 K 2023-04-12 13:54:31,508 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/24d8ab71f01c4740a6331035f0e319eb as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/24d8ab71f01c4740a6331035f0e319eb 2023-04-12 13:54:31,518 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/24d8ab71f01c4740a6331035f0e319eb, entries=150, sequenceid=272, filesize=11.9 K 2023-04-12 13:54:31,521 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6b28deeffcdc135c24d47af5464730df in 873ms, sequenceid=272, compaction requested=true 2023-04-12 13:54:31,521 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:31,521 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:31,523 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:31,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:31,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66353b8e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:31,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:31,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:31,527 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,527 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:31,530 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:31,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:31,533 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:31,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:31,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:31,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:31,533 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:31,537 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,537 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:31,537 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:31,539 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,541 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:31,542 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:31,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:31,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:31,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:31,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,542 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:31,542 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:31,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:31,642 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:31,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:31,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:31,646 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:31,646 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:31,646 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:31,646 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:31,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:31,648 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:31,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:31,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:31,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:31,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:31,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:31,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:31,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:31,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:31,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. started... 2023-04-12 13:54:31,651 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:31,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:31,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:31,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:31,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:31,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:31,651 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:31,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:31,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. as already flushing 2023-04-12 13:54:31,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307731822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307731823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307731824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307731824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307731825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,900 DEBUG [Thread-872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:58527 2023-04-12 13:54:31,900 DEBUG [Thread-870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:58527 2023-04-12 13:54:31,900 DEBUG [Thread-872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:31,901 DEBUG [Thread-870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:31,902 DEBUG [Thread-868] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:58527 2023-04-12 13:54:31,902 DEBUG [Thread-868] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:31,905 DEBUG [Thread-874] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:58527 2023-04-12 13:54:31,905 DEBUG [Thread-874] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:31,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307731926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307731930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307731930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307731930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:31,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:31,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307731930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,098 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041264186996918e475eb0c1d03da152075c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041264186996918e475eb0c1d03da152075c 2023-04-12 13:54:32,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307732128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307732131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307732132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307732132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307732132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307732432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307732433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307732434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307732434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307732434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,505 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/49e19b0f2eeb449787003e19866272d2 2023-04-12 13:54:32,915 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/8c721b7472124f1887ae2ce12dd7f6e5 2023-04-12 13:54:32,934 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/66f2577f947146f5b8959d2b3f0ba589 2023-04-12 13:54:32,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36132 deadline: 1681307732934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36118 deadline: 1681307732936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36176 deadline: 1681307732936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36160 deadline: 1681307732938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:32,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36108 deadline: 1681307732938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,939 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/49e19b0f2eeb449787003e19866272d2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/49e19b0f2eeb449787003e19866272d2 2023-04-12 13:54:32,944 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/49e19b0f2eeb449787003e19866272d2, entries=100, sequenceid=285, filesize=18.6 K 2023-04-12 13:54:32,945 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/8c721b7472124f1887ae2ce12dd7f6e5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/8c721b7472124f1887ae2ce12dd7f6e5 2023-04-12 13:54:32,949 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/8c721b7472124f1887ae2ce12dd7f6e5, entries=100, sequenceid=285, filesize=9.5 K 2023-04-12 13:54:32,950 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/66f2577f947146f5b8959d2b3f0ba589 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/66f2577f947146f5b8959d2b3f0ba589 2023-04-12 13:54:32,954 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/66f2577f947146f5b8959d2b3f0ba589, entries=100, sequenceid=285, filesize=9.5 K 2023-04-12 13:54:32,954 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6b28deeffcdc135c24d47af5464730df in 1303ms, sequenceid=285, compaction requested=true 2023-04-12 13:54:32,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:32,955 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:32,956 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,956 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,957 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:32,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b64f965[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:32,958 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:32,958 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,959 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:32,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:32,961 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:32,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:32,963 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:32,963 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:32,963 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:32,963 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:32,963 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:32,965 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,965 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,965 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:32,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:32,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:32,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:32,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:32,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:32,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:32,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:32,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:32,969 DEBUG [Thread-867] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,969 DEBUG [Thread-867] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:32,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:33,069 DEBUG [Thread-867] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:33,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:33,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:33,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=114.05 KB heapSize=339.89 KB 2023-04-12 13:54:33,940 DEBUG [Thread-859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:58527 2023-04-12 13:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:33,940 DEBUG [Thread-863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:58527 2023-04-12 13:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:33,940 DEBUG [Thread-859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:33,940 DEBUG [Thread-863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:33,940 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:33,944 DEBUG [Thread-857] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:58527 2023-04-12 13:54:33,945 DEBUG [Thread-857] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:33,945 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:58527 2023-04-12 13:54:33,945 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:33,950 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:58527 2023-04-12 13:54:33,950 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5487 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5690 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 2316 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 6947 rows 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 2320 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 6959 rows 2023-04-12 13:54:33,950 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:54:33,951 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:58527 2023-04-12 13:54:33,951 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:33,952 INFO [Listener at localhost/46495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 13:54:33,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 13:54:33,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:33,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:33,957 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307673957"}]},"ts":"1681307673957"} 2023-04-12 13:54:33,958 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 13:54:33,961 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 13:54:33,961 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, UNASSIGN}] 2023-04-12 13:54:33,963 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, UNASSIGN 2023-04-12 13:54:33,963 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:33,964 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307673963"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307673963"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307673963"}]},"ts":"1681307673963"} 2023-04-12 13:54:33,964 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:54:34,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:34,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:34,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6b28deeffcdc135c24d47af5464730df, disabling compactions & flushes 2023-04-12 13:54:34,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:34,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:34,355 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304122a44b69e097249d59db2a1433d51348c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122a44b69e097249d59db2a1433d51348c 2023-04-12 13:54:34,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:34,760 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/f20364f77d0b4ef2854436ebb5952beb 2023-04-12 13:54:35,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:35,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/d0558d215ec54239adcba2ffb229b79c 2023-04-12 13:54:35,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/fb9738d48c1f4b5db885dc464b230eb1 2023-04-12 13:54:35,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/f20364f77d0b4ef2854436ebb5952beb as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f20364f77d0b4ef2854436ebb5952beb 2023-04-12 13:54:35,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f20364f77d0b4ef2854436ebb5952beb, entries=150, sequenceid=306, filesize=25.4 K 2023-04-12 13:54:35,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/d0558d215ec54239adcba2ffb229b79c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/d0558d215ec54239adcba2ffb229b79c 2023-04-12 13:54:35,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/d0558d215ec54239adcba2ffb229b79c, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:54:35,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/fb9738d48c1f4b5db885dc464b230eb1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/fb9738d48c1f4b5db885dc464b230eb1 2023-04-12 13:54:35,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/fb9738d48c1f4b5db885dc464b230eb1, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:54:35,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for 6b28deeffcdc135c24d47af5464730df in 1664ms, sequenceid=306, compaction requested=true 2023-04-12 13:54:35,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:35,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:35,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:35,604 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. because compaction request was cancelled 2023-04-12 13:54:35,604 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. because compaction request was cancelled 2023-04-12 13:54:35,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:35,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:35,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. after waiting 0 ms 2023-04-12 13:54:35,604 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. because compaction request was cancelled 2023-04-12 13:54:35,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:35,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:35,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6b28deeffcdc135c24d47af5464730df 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-12 13:54:35,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=A 2023-04-12 13:54:35,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:35,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=B 2023-04-12 13:54:35,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:35,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6b28deeffcdc135c24d47af5464730df, store=C 2023-04-12 13:54:35,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:36,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304126a05db83cf0f4ea5aa0910176af065e0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304126a05db83cf0f4ea5aa0910176af065e0 2023-04-12 13:54:36,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:36,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/d22ee722a9904cb5a69fbc9422ff2ace 2023-04-12 13:54:36,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c33bcbf06a9a41909ab6d20bdd651b94 2023-04-12 13:54:37,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/9eecc5423389472cb96e54f90b7e5e83 2023-04-12 13:54:37,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/A/d22ee722a9904cb5a69fbc9422ff2ace as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d22ee722a9904cb5a69fbc9422ff2ace 2023-04-12 13:54:37,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d22ee722a9904cb5a69fbc9422ff2ace, entries=50, sequenceid=312, filesize=11.7 K 2023-04-12 13:54:37,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/B/c33bcbf06a9a41909ab6d20bdd651b94 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c33bcbf06a9a41909ab6d20bdd651b94 2023-04-12 13:54:37,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c33bcbf06a9a41909ab6d20bdd651b94, entries=50, sequenceid=312, filesize=7.1 K 2023-04-12 13:54:37,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/.tmp/C/9eecc5423389472cb96e54f90b7e5e83 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/9eecc5423389472cb96e54f90b7e5e83 2023-04-12 13:54:37,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/9eecc5423389472cb96e54f90b7e5e83, entries=50, sequenceid=312, filesize=7.1 K 2023-04-12 13:54:37,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 6b28deeffcdc135c24d47af5464730df in 1661ms, sequenceid=312, compaction requested=true 2023-04-12 13:54:37,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2c62f366ca204f88a2dcb507b2630439, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/46d8e43d4bb34293aaf80bdec48b3fb3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f4a521149fd741208a0400c6b33ab3cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/75ac9fcc41bc445180fc99aa90710b94, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a551f3514db546639650d4b8093827fa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/66798c05116e420e81efdca53e4881c9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/79353a99c5ee4f6eb0cb5f186c7f4a5f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/3d0eb6ace92a413581b8a7e108315be4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/06b7e095c32a41e59753a80bbdb38f00, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/92975ac385404ba580e51b7a806af7af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/835bbe12afb245bc803c4fb3facdc758, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d47133d15ac1406d99c7c9f6551d7f84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/e3846261dbfc48759e1937e7d236213e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2a1c2715fab44cfbb82236b6034afb6c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/192aa925683148d49442ce09d49b78ee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/6507df9b08c54b8f9e1d17aa3acd2680] to archive 2023-04-12 13:54:37,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:54:37,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2c62f366ca204f88a2dcb507b2630439 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2c62f366ca204f88a2dcb507b2630439 2023-04-12 13:54:37,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/46d8e43d4bb34293aaf80bdec48b3fb3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/46d8e43d4bb34293aaf80bdec48b3fb3 2023-04-12 13:54:37,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f4a521149fd741208a0400c6b33ab3cf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f4a521149fd741208a0400c6b33ab3cf 2023-04-12 13:54:37,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/75ac9fcc41bc445180fc99aa90710b94 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/75ac9fcc41bc445180fc99aa90710b94 2023-04-12 13:54:37,288 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a551f3514db546639650d4b8093827fa to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a551f3514db546639650d4b8093827fa 2023-04-12 13:54:37,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/66798c05116e420e81efdca53e4881c9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/66798c05116e420e81efdca53e4881c9 2023-04-12 13:54:37,290 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/79353a99c5ee4f6eb0cb5f186c7f4a5f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/79353a99c5ee4f6eb0cb5f186c7f4a5f 2023-04-12 13:54:37,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/3d0eb6ace92a413581b8a7e108315be4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/3d0eb6ace92a413581b8a7e108315be4 2023-04-12 13:54:37,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/06b7e095c32a41e59753a80bbdb38f00 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/06b7e095c32a41e59753a80bbdb38f00 2023-04-12 13:54:37,292 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/92975ac385404ba580e51b7a806af7af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/92975ac385404ba580e51b7a806af7af 2023-04-12 13:54:37,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/835bbe12afb245bc803c4fb3facdc758 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/835bbe12afb245bc803c4fb3facdc758 2023-04-12 13:54:37,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d47133d15ac1406d99c7c9f6551d7f84 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d47133d15ac1406d99c7c9f6551d7f84 2023-04-12 13:54:37,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/e3846261dbfc48759e1937e7d236213e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/e3846261dbfc48759e1937e7d236213e 2023-04-12 13:54:37,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2a1c2715fab44cfbb82236b6034afb6c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/2a1c2715fab44cfbb82236b6034afb6c 2023-04-12 13:54:37,297 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/192aa925683148d49442ce09d49b78ee to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/192aa925683148d49442ce09d49b78ee 2023-04-12 13:54:37,298 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/6507df9b08c54b8f9e1d17aa3acd2680 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/6507df9b08c54b8f9e1d17aa3acd2680 2023-04-12 13:54:37,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/ab7a8bf15b0d4128a5b462726518b571, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ef84ae724c44baf9cc39a53ebf5b528, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7d12f59ef2af4b62b78c9e884f24212a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c245e7b52b564d26bc74be17bab2c2c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/dbcf2a03da6a4a89a9148afc76a9dd74, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e92af0ba32e84b5aab581839978ce4da, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/15f7b0e807c9456fbd118bf273758f5b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/151936e2e3194e7eba53639de080950a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ffe2e3d38284cc4bc9964c6848c3d63, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e1f26133683c4e5e8020f9911a696417, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c654c02ebb784de49e838f0b562743de, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/833310f730674389a8b5298f3dac39d6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/45f05efa74594a5c847fa8bfd6b5266c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c3462217efe54094bc7017f164ef357b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e00066a2b167411982e00a35884227af] to archive 2023-04-12 13:54:37,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:54:37,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/ab7a8bf15b0d4128a5b462726518b571 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/ab7a8bf15b0d4128a5b462726518b571 2023-04-12 13:54:37,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ef84ae724c44baf9cc39a53ebf5b528 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ef84ae724c44baf9cc39a53ebf5b528 2023-04-12 13:54:37,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7d12f59ef2af4b62b78c9e884f24212a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7d12f59ef2af4b62b78c9e884f24212a 2023-04-12 13:54:37,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c245e7b52b564d26bc74be17bab2c2c8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c245e7b52b564d26bc74be17bab2c2c8 2023-04-12 13:54:37,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/dbcf2a03da6a4a89a9148afc76a9dd74 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/dbcf2a03da6a4a89a9148afc76a9dd74 2023-04-12 13:54:37,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e92af0ba32e84b5aab581839978ce4da to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e92af0ba32e84b5aab581839978ce4da 2023-04-12 13:54:37,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/a9e682d0a0ee4bf0b6754d5c20ed4fc3 2023-04-12 13:54:37,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/15f7b0e807c9456fbd118bf273758f5b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/15f7b0e807c9456fbd118bf273758f5b 2023-04-12 13:54:37,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/151936e2e3194e7eba53639de080950a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/151936e2e3194e7eba53639de080950a 2023-04-12 13:54:37,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ffe2e3d38284cc4bc9964c6848c3d63 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/7ffe2e3d38284cc4bc9964c6848c3d63 2023-04-12 13:54:37,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e1f26133683c4e5e8020f9911a696417 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e1f26133683c4e5e8020f9911a696417 2023-04-12 13:54:37,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c654c02ebb784de49e838f0b562743de to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c654c02ebb784de49e838f0b562743de 2023-04-12 13:54:37,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/833310f730674389a8b5298f3dac39d6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/833310f730674389a8b5298f3dac39d6 2023-04-12 13:54:37,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/45f05efa74594a5c847fa8bfd6b5266c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/45f05efa74594a5c847fa8bfd6b5266c 2023-04-12 13:54:37,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c3462217efe54094bc7017f164ef357b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c3462217efe54094bc7017f164ef357b 2023-04-12 13:54:37,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e00066a2b167411982e00a35884227af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/e00066a2b167411982e00a35884227af 2023-04-12 13:54:37,325 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a8d412e0f2a64ec784292fa2263289d7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67a8977a98bc4588ba389ea40daa1c67, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/4602bf5f67794e5493e2a3e24d8d4005, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/90c839c7cbda4e1aaba7639df17b45fe, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/cc74dc428ac3493ebce8bb57fca33e8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/3373698d88fd4037ac289fc2aab41423, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/42cb16d50c034930adf8e3cd38beff70, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/70390f1699ff400e9c8556b15dc9ce54, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/8ba4f4f61c9b48a79e54a38a3274f055, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/187806e62e4e4089b2fda2fc8d5dd5cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a93e54c972cd4ebaa6c3f8aa90f377f7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/2e65cbf4ff224c33a05f8d4098199732, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67b48a55ab3e4ab8b9471582b72d8c6d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/210092fc6c88419b85739dea6cb0cc19, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/e51db76332c94a44ac279f34bc1e6710, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/796f7a714e444059b6b4e5efd8e31225] to archive 2023-04-12 13:54:37,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:54:37,327 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a8d412e0f2a64ec784292fa2263289d7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a8d412e0f2a64ec784292fa2263289d7 2023-04-12 13:54:37,328 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67a8977a98bc4588ba389ea40daa1c67 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67a8977a98bc4588ba389ea40daa1c67 2023-04-12 13:54:37,329 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/4602bf5f67794e5493e2a3e24d8d4005 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/4602bf5f67794e5493e2a3e24d8d4005 2023-04-12 13:54:37,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/90c839c7cbda4e1aaba7639df17b45fe to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/90c839c7cbda4e1aaba7639df17b45fe 2023-04-12 13:54:37,331 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/cc74dc428ac3493ebce8bb57fca33e8d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/cc74dc428ac3493ebce8bb57fca33e8d 2023-04-12 13:54:37,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/3373698d88fd4037ac289fc2aab41423 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/3373698d88fd4037ac289fc2aab41423 2023-04-12 13:54:37,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/42cb16d50c034930adf8e3cd38beff70 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/42cb16d50c034930adf8e3cd38beff70 2023-04-12 13:54:37,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/70390f1699ff400e9c8556b15dc9ce54 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/70390f1699ff400e9c8556b15dc9ce54 2023-04-12 13:54:37,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/8ba4f4f61c9b48a79e54a38a3274f055 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/8ba4f4f61c9b48a79e54a38a3274f055 2023-04-12 13:54:37,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/187806e62e4e4089b2fda2fc8d5dd5cf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/187806e62e4e4089b2fda2fc8d5dd5cf 2023-04-12 13:54:37,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a93e54c972cd4ebaa6c3f8aa90f377f7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/a93e54c972cd4ebaa6c3f8aa90f377f7 2023-04-12 13:54:37,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/2e65cbf4ff224c33a05f8d4098199732 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/2e65cbf4ff224c33a05f8d4098199732 2023-04-12 13:54:37,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67b48a55ab3e4ab8b9471582b72d8c6d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/67b48a55ab3e4ab8b9471582b72d8c6d 2023-04-12 13:54:37,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/210092fc6c88419b85739dea6cb0cc19 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/210092fc6c88419b85739dea6cb0cc19 2023-04-12 13:54:37,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/e51db76332c94a44ac279f34bc1e6710 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/e51db76332c94a44ac279f34bc1e6710 2023-04-12 13:54:37,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/796f7a714e444059b6b4e5efd8e31225 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/796f7a714e444059b6b4e5efd8e31225 2023-04-12 13:54:37,347 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/recovered.edits/315.seqid, newMaxSeqId=315, maxSeqId=4 2023-04-12 13:54:37,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df. 2023-04-12 13:54:37,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6b28deeffcdc135c24d47af5464730df: 2023-04-12 13:54:37,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:37,350 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=6b28deeffcdc135c24d47af5464730df, regionState=CLOSED 2023-04-12 13:54:37,350 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307677350"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307677350"}]},"ts":"1681307677350"} 2023-04-12 13:54:37,352 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-12 13:54:37,352 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 6b28deeffcdc135c24d47af5464730df, server=jenkins-hbase4.apache.org,35663,1681307616979 in 3.3870 sec 2023-04-12 13:54:37,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-12 13:54:37,354 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6b28deeffcdc135c24d47af5464730df, UNASSIGN in 3.3910 sec 2023-04-12 13:54:37,354 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307677354"}]},"ts":"1681307677354"} 2023-04-12 13:54:37,355 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 13:54:37,356 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 13:54:37,358 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.4040 sec 2023-04-12 13:54:38,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 13:54:38,063 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-12 13:54:38,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 13:54:38,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,066 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-12 13:54:38,066 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,071 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:38,074 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/recovered.edits] 2023-04-12 13:54:38,077 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/49e19b0f2eeb449787003e19866272d2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/49e19b0f2eeb449787003e19866272d2 2023-04-12 13:54:38,078 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a8ceb17e14de44158c43b6a2a1a9c3f7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/a8ceb17e14de44158c43b6a2a1a9c3f7 2023-04-12 13:54:38,079 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/caa63b76c2cf4fe2941c594a46806789 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/caa63b76c2cf4fe2941c594a46806789 2023-04-12 13:54:38,081 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/cb2396cba9114b42a69b054cda9e4db5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/cb2396cba9114b42a69b054cda9e4db5 2023-04-12 13:54:38,082 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d22ee722a9904cb5a69fbc9422ff2ace to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/d22ee722a9904cb5a69fbc9422ff2ace 2023-04-12 13:54:38,083 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f20364f77d0b4ef2854436ebb5952beb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/A/f20364f77d0b4ef2854436ebb5952beb 2023-04-12 13:54:38,085 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/4146d79ca02247c99fdbd80b52498ad2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/4146d79ca02247c99fdbd80b52498ad2 2023-04-12 13:54:38,087 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/8c721b7472124f1887ae2ce12dd7f6e5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/8c721b7472124f1887ae2ce12dd7f6e5 2023-04-12 13:54:38,088 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/b9683664dd5d4ca49ff179d65d6548ce to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/b9683664dd5d4ca49ff179d65d6548ce 2023-04-12 13:54:38,089 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c33bcbf06a9a41909ab6d20bdd651b94 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/c33bcbf06a9a41909ab6d20bdd651b94 2023-04-12 13:54:38,090 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/d0558d215ec54239adcba2ffb229b79c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/d0558d215ec54239adcba2ffb229b79c 2023-04-12 13:54:38,091 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/efbf86442d624787acbd0cb86e5bd889 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/B/efbf86442d624787acbd0cb86e5bd889 2023-04-12 13:54:38,093 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/24d8ab71f01c4740a6331035f0e319eb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/24d8ab71f01c4740a6331035f0e319eb 2023-04-12 13:54:38,094 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/66f2577f947146f5b8959d2b3f0ba589 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/66f2577f947146f5b8959d2b3f0ba589 2023-04-12 13:54:38,095 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/715cc48d6088425e9c1a6a0bd0d2d3c1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/715cc48d6088425e9c1a6a0bd0d2d3c1 2023-04-12 13:54:38,097 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/9eecc5423389472cb96e54f90b7e5e83 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/9eecc5423389472cb96e54f90b7e5e83 2023-04-12 13:54:38,098 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/f7a92c9dbd154663a779dfbc11aae3df to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/f7a92c9dbd154663a779dfbc11aae3df 2023-04-12 13:54:38,099 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/fb9738d48c1f4b5db885dc464b230eb1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/C/fb9738d48c1f4b5db885dc464b230eb1 2023-04-12 13:54:38,102 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/recovered.edits/315.seqid to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df/recovered.edits/315.seqid 2023-04-12 13:54:38,103 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/6b28deeffcdc135c24d47af5464730df 2023-04-12 13:54:38,103 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:54:38,103 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 13:54:38,104 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-12 13:54:38,107 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122a44b69e097249d59db2a1433d51348c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122a44b69e097249d59db2a1433d51348c 2023-04-12 13:54:38,108 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122c9ec9bac4a2470a904a7081a9c1c007 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122c9ec9bac4a2470a904a7081a9c1c007 2023-04-12 13:54:38,109 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122fd09a5c47754b8e9687b8a8295eeaf9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122fd09a5c47754b8e9687b8a8295eeaf9 2023-04-12 13:54:38,110 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123807c9f8604c40f18cfe4997e8b25677 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123807c9f8604c40f18cfe4997e8b25677 2023-04-12 13:54:38,111 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041238bf7a5b28b645aab59e997368e0d670 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041238bf7a5b28b645aab59e997368e0d670 2023-04-12 13:54:38,112 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124762fd370a134d86bf66ea6b5e41b7e4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124762fd370a134d86bf66ea6b5e41b7e4 2023-04-12 13:54:38,114 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041264186996918e475eb0c1d03da152075c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041264186996918e475eb0c1d03da152075c 2023-04-12 13:54:38,115 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304126a05db83cf0f4ea5aa0910176af065e0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304126a05db83cf0f4ea5aa0910176af065e0 2023-04-12 13:54:38,116 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127d8e39e1d8b14b53ae651d8ec71cb970 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127d8e39e1d8b14b53ae651d8ec71cb970 2023-04-12 13:54:38,117 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e1ed45e69be48579cf8eea488539621 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127e1ed45e69be48579cf8eea488539621 2023-04-12 13:54:38,118 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041298ebbb66ad044668b607acbd237af051 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041298ebbb66ad044668b607acbd237af051 2023-04-12 13:54:38,119 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b0593c8a25844a8c9edbf910c2514f9c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b0593c8a25844a8c9edbf910c2514f9c 2023-04-12 13:54:38,120 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b557be98db4f4e74a2ad1bf2b744fbc0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b557be98db4f4e74a2ad1bf2b744fbc0 2023-04-12 13:54:38,121 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c157233e36de46d086f27faf0d1ef2ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412c157233e36de46d086f27faf0d1ef2ab 2023-04-12 13:54:38,122 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d10837337f2640d48211c43a377625db to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d10837337f2640d48211c43a377625db 2023-04-12 13:54:38,124 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412dad6ba252b194c93bde768291c5ed206 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412dad6ba252b194c93bde768291c5ed206 2023-04-12 13:54:38,125 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e0856cf8913244238a2154a2f15d4109 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e0856cf8913244238a2154a2f15d4109 2023-04-12 13:54:38,126 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f38593c7ced04e4bb5cfb2488a6eb1c0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f38593c7ced04e4bb5cfb2488a6eb1c0 2023-04-12 13:54:38,127 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fce95721f10d4243938c4408f341e573 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fce95721f10d4243938c4408f341e573 2023-04-12 13:54:38,127 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 13:54:38,130 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,132 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 13:54:38,134 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 13:54:38,135 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,135 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 13:54:38,135 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681307678135"}]},"ts":"9223372036854775807"} 2023-04-12 13:54:38,137 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 13:54:38,137 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6b28deeffcdc135c24d47af5464730df, NAME => 'TestAcidGuarantees,,1681307648732.6b28deeffcdc135c24d47af5464730df.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 13:54:38,137 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 13:54:38,137 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681307678137"}]},"ts":"9223372036854775807"} 2023-04-12 13:54:38,139 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 13:54:38,140 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,141 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 78 msec 2023-04-12 13:54:38,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-12 13:54:38,167 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-12 13:54:38,180 INFO [Listener at localhost/46495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233 (was 234), OpenFileDescriptor=547 (was 563), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=484 (was 451) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=389 (was 521) 2023-04-12 13:54:38,193 INFO [Listener at localhost/46495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=233, OpenFileDescriptor=547, MaxFileDescriptor=60000, SystemLoadAverage=484, ProcessCount=169, AvailableMemoryMB=388 2023-04-12 13:54:38,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:54:38,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:54:38,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 13:54:38,197 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:54:38,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-12 13:54:38,198 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:54:38,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 13:54:38,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a empty. 2023-04-12 13:54:38,200 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,201 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:54:38,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 13:54:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 13:54:38,609 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:54:38,611 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8e79b7aef32bdb05b28a7dce92e13c3a, NAME => 'TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:54:38,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:54:38,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8e79b7aef32bdb05b28a7dce92e13c3a, disabling compactions & flushes 2023-04-12 13:54:38,616 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. after waiting 0 ms 2023-04-12 13:54:38,616 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,617 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:38,618 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:54:38,619 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307678619"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307678619"}]},"ts":"1681307678619"} 2023-04-12 13:54:38,620 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:54:38,621 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:54:38,621 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307678621"}]},"ts":"1681307678621"} 2023-04-12 13:54:38,621 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 13:54:38,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, ASSIGN}] 2023-04-12 13:54:38,625 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, ASSIGN 2023-04-12 13:54:38,626 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:54:38,777 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=8e79b7aef32bdb05b28a7dce92e13c3a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:38,777 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307678777"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307678777"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307678777"}]},"ts":"1681307678777"} 2023-04-12 13:54:38,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:54:38,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 13:54:38,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8e79b7aef32bdb05b28a7dce92e13c3a, NAME => 'TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:54:38,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:54:38,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,935 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,937 DEBUG [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A 2023-04-12 13:54:38,937 DEBUG [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A 2023-04-12 13:54:38,937 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:38,937 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8e79b7aef32bdb05b28a7dce92e13c3a columnFamilyName A 2023-04-12 13:54:38,938 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.HStore(310): Store=8e79b7aef32bdb05b28a7dce92e13c3a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:38,938 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,939 DEBUG [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B 2023-04-12 13:54:38,939 DEBUG [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B 2023-04-12 13:54:38,939 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:38,939 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8e79b7aef32bdb05b28a7dce92e13c3a columnFamilyName B 2023-04-12 13:54:38,940 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.HStore(310): Store=8e79b7aef32bdb05b28a7dce92e13c3a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:38,940 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,941 DEBUG [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C 2023-04-12 13:54:38,941 DEBUG [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C 2023-04-12 13:54:38,941 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:54:38,941 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8e79b7aef32bdb05b28a7dce92e13c3a columnFamilyName C 2023-04-12 13:54:38,942 INFO [StoreOpener-8e79b7aef32bdb05b28a7dce92e13c3a-1] regionserver.HStore(310): Store=8e79b7aef32bdb05b28a7dce92e13c3a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:54:38,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:54:38,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:38,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:54:38,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8e79b7aef32bdb05b28a7dce92e13c3a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69865914, jitterRate=0.04108324646949768}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:54:38,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:38,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., pid=30, masterSystemTime=1681307678930 2023-04-12 13:54:38,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:38,950 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=8e79b7aef32bdb05b28a7dce92e13c3a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:38,950 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307678950"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307678950"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307678950"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307678950"}]},"ts":"1681307678950"} 2023-04-12 13:54:38,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-12 13:54:38,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 in 172 msec 2023-04-12 13:54:38,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-12 13:54:38,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, ASSIGN in 328 msec 2023-04-12 13:54:38,955 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:54:38,955 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307678955"}]},"ts":"1681307678955"} 2023-04-12 13:54:38,956 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 13:54:38,958 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:54:38,960 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 764 msec 2023-04-12 13:54:39,158 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 13:54:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-12 13:54:39,302 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-12 13:54:39,303 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,308 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,309 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,310 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,311 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:54:39,314 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:54:39,315 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,319 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,319 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,322 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,323 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,327 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,327 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,331 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,331 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,334 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,335 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,343 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,343 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,347 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,347 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,351 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,351 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,357 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,357 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:54:39,361 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:54:39,365 DEBUG [hconnection-0x55282fc1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,365 DEBUG [hconnection-0x1ee1834d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,366 DEBUG [hconnection-0x48d1bd85-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,366 DEBUG [hconnection-0x6b2e2b79-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,366 DEBUG [hconnection-0x52692149-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:39,367 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41512, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,367 DEBUG [hconnection-0x44174c30-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,367 DEBUG [hconnection-0x626dcdd4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,368 DEBUG [hconnection-0x59d7658c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,368 DEBUG [hconnection-0x75eb7907-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,368 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,368 DEBUG [hconnection-0x467a7e52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:54:39,370 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,371 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,371 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41532, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,375 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41538, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41520, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,375 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,376 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:54:39,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:39,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:39,382 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:39,382 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,382 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:39,382 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:39,382 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,382 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,384 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,384 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,384 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,384 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:39,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:39,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:39,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:39,392 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:39,393 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:39,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:39,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:54:39,393 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,394 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:39,394 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:39,395 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:39,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:39,395 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,395 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:39,395 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:39,395 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:39,395 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,395 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:39,397 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:39,397 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:39,397 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:39,397 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:39,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,405 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:39,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:39,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e9f446a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:39,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:39,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,410 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,410 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:39,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:39,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:39,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:39,412 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:39,415 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:39,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:39,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:39,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:39,415 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:39,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307679408, max timestamp=1681307679414Num uniques 859; 2023-04-12 13:54:39,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:39,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307679408, max timestamp=1681307679414Num uniques 859; 2023-04-12 13:54:39,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:39,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307679408, max timestamp=1681307679414Num uniques 859; 2023-04-12 13:54:39,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:39,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,423 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:39,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:39,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,424 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:39,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,424 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:39,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307739448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307739452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307739454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307739454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307739454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,524 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:39,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:39,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:39,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:39,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:39,533 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:39,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:39,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:39,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,535 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,536 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,536 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:39,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:39,544 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:39,544 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,544 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:39,544 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:39,544 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:39,544 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,544 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:39,545 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:39,545 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:39,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:39,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:39,545 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:39,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:39,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766a3a54[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:39,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:39,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,550 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:39,550 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:39,550 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:39,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,552 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:39,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:39,555 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:39,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:39,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:39,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:39,556 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:39,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307739555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307739556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,558 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,558 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,558 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307739558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307739559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307739559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,563 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:39,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:39,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,564 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,564 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,564 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,664 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:39,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:39,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:39,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:39,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:39,667 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:39,667 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,667 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:39,667 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:39,668 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,668 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,676 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,676 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,676 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,676 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:39,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:39,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:39,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:39,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:39,678 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,678 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:39,678 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:39,678 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:39,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,680 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:39,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39cc865c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:39,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:39,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,683 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:39,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:39,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:39,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:39,687 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:39,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:39,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:39,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:39,687 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:39,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,690 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,690 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:39,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,696 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,696 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:39,696 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:39,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:39,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307739758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307739759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307739761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307739761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:39,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307739763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,797 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:39,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:39,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:39,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:39,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:39,800 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:39,800 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:39,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:39,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,802 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:39,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:39,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:39,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:39,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:39,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:39,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:39,806 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:39,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35efefd1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:39,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:39,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:39,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:39,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:39,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:39,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:39,813 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:39,813 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:39,813 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:39,813 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:39,813 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:39,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,815 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,816 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,818 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,818 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,818 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:39,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:39,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,819 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,819 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:39,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5a47ea6879284a9180281b69ba870147 2023-04-12 13:54:39,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/fa67792fef1e432ead4960ab03e31653 2023-04-12 13:54:39,919 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:39,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:39,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:39,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:39,922 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:39,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:39,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:39,923 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,923 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,926 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,926 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,927 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,927 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:39,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:39,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:39,928 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:39,928 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:39,928 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:39,928 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:39,928 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:39,928 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,928 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:39,929 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:39,929 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:39,929 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:39,930 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,930 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:39,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f74112[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:39,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:39,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,933 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:39,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:39,933 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,933 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:39,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:39,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:39,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:39,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:39,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:39,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:39,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:39,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:39,940 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,940 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:39,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,940 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,941 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:39,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:39,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:39,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:39,944 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:39,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:39,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:39,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:39,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:39,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:39,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:39,945 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:39,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:39,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:39,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:39,945 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:39,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:39,945 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,045 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:40,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:40,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:40,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:40,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:40,049 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:40,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:40,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:40,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:40,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,051 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:40,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:40,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:40,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:40,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:40,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:40,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:40,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:40,053 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:40,053 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:40,053 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:40,053 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:40,053 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:40,054 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:40,056 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:40,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:40,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:40,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36e871e2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:40,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:40,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307740061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307740062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,064 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:40,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:40,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:40,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:40,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:40,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:40,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:40,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307740066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:40,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:40,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307740066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:40,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307740067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:40,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:40,069 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:40,069 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:40,069 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:40,069 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:40,069 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:40,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:40,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:40,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:40,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:40,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:40,076 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:40,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:40,076 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:40,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,076 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:40,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:40,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:40,177 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:40,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:40,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:40,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:40,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:40,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:40,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:40,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:40,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:40,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:40,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:40,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:40,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:40,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:40,187 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:40,187 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:40,187 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:40,187 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:40,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:40,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:40,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:40,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:40,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:40,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:40,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bcbef05[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:40,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:40,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,194 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,194 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:40,194 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:40,194 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:40,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:40,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,195 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:40,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:40,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:40,195 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:40,195 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:40,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:40,198 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:40,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:40,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:40,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:40,198 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:40,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:40,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:40,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,202 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:40,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:40,205 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:40,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:40,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:40,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:40,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:40,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,205 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:40,205 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:40,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:40,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:40,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/c57600d9ef8d42c1a32eaf1ed8bca5e4 2023-04-12 13:54:40,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5a47ea6879284a9180281b69ba870147 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a47ea6879284a9180281b69ba870147 2023-04-12 13:54:40,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a47ea6879284a9180281b69ba870147, entries=100, sequenceid=12, filesize=9.3 K 2023-04-12 13:54:40,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/fa67792fef1e432ead4960ab03e31653 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/fa67792fef1e432ead4960ab03e31653 2023-04-12 13:54:40,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/fa67792fef1e432ead4960ab03e31653, entries=100, sequenceid=12, filesize=9.3 K 2023-04-12 13:54:40,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/c57600d9ef8d42c1a32eaf1ed8bca5e4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/c57600d9ef8d42c1a32eaf1ed8bca5e4 2023-04-12 13:54:40,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/c57600d9ef8d42c1a32eaf1ed8bca5e4, entries=100, sequenceid=12, filesize=9.3 K 2023-04-12 13:54:40,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 8e79b7aef32bdb05b28a7dce92e13c3a in 907ms, sequenceid=12, compaction requested=false 2023-04-12 13:54:40,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:40,306 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:40,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:40,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:40,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:40,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:40,309 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:40,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:40,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:40,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:40,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:40,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:40,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:40,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:40,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:40,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:40,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:40,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:40,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:40,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:40,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:40,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:40,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:40,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:40,312 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:40,312 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:40,313 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-12 13:54:40,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:40,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:40,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:40,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:40,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:40,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:40,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:40,568 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:40,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307740572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307740572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307740574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307740574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307740575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307740675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307740677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307740678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307740678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,739 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5a85179c8598419d8da9c404a0a9e0aa 2023-04-12 13:54:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307740879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307740879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307740882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:40,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:40,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307740882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,152 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/afc0c108c5f64a54b414656b4f5d241b 2023-04-12 13:54:41,165 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b82af56af245491eb05943713c60ae65 2023-04-12 13:54:41,169 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5a85179c8598419d8da9c404a0a9e0aa as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a85179c8598419d8da9c404a0a9e0aa 2023-04-12 13:54:41,173 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a85179c8598419d8da9c404a0a9e0aa, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:54:41,174 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/afc0c108c5f64a54b414656b4f5d241b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/afc0c108c5f64a54b414656b4f5d241b 2023-04-12 13:54:41,178 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/afc0c108c5f64a54b414656b4f5d241b, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:54:41,179 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b82af56af245491eb05943713c60ae65 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b82af56af245491eb05943713c60ae65 2023-04-12 13:54:41,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,183 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b82af56af245491eb05943713c60ae65, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:54:41,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307741182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,184 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 8e79b7aef32bdb05b28a7dce92e13c3a in 872ms, sequenceid=49, compaction requested=false 2023-04-12 13:54:41,184 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:41,184 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:41,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:41,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:41,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:41,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211004ca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:41,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:41,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:41,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:41,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:41,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:41,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:41,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:41,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:41,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:41,193 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:41,193 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,193 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:41,193 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:41,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:41,196 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:41,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:41,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:41,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:41,196 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:41,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:41,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:41,199 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/3bc52315936c4ef29634df7ceec2e491 2023-04-12 13:54:41,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:41,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:41,203 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:41,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:41,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:41,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:41,204 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:41,204 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:41,204 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:41,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:41,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307741213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307741217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307741217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/ba993c300de14cf285f8f16a93af4b94 2023-04-12 13:54:41,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/37fffa21ff0f43ffbf84d55e9eac0577 2023-04-12 13:54:41,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/3bc52315936c4ef29634df7ceec2e491 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3bc52315936c4ef29634df7ceec2e491 2023-04-12 13:54:41,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3bc52315936c4ef29634df7ceec2e491, entries=150, sequenceid=62, filesize=11.6 K 2023-04-12 13:54:41,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/ba993c300de14cf285f8f16a93af4b94 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba993c300de14cf285f8f16a93af4b94 2023-04-12 13:54:41,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba993c300de14cf285f8f16a93af4b94, entries=150, sequenceid=62, filesize=11.6 K 2023-04-12 13:54:41,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/37fffa21ff0f43ffbf84d55e9eac0577 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/37fffa21ff0f43ffbf84d55e9eac0577 2023-04-12 13:54:41,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/37fffa21ff0f43ffbf84d55e9eac0577, entries=150, sequenceid=62, filesize=11.6 K 2023-04-12 13:54:41,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 8e79b7aef32bdb05b28a7dce92e13c3a in 78ms, sequenceid=62, compaction requested=true 2023-04-12 13:54:41,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:41,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:41,265 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:41,265 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:41,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:41,265 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:41,266 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:41,266 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:41,267 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:41,267 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/fa67792fef1e432ead4960ab03e31653, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/afc0c108c5f64a54b414656b4f5d241b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba993c300de14cf285f8f16a93af4b94] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=32.5 K 2023-04-12 13:54:41,267 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:41,267 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:41,267 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting fa67792fef1e432ead4960ab03e31653, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681307679386 2023-04-12 13:54:41,267 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:41,267 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a47ea6879284a9180281b69ba870147, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a85179c8598419d8da9c404a0a9e0aa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3bc52315936c4ef29634df7ceec2e491] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=32.5 K 2023-04-12 13:54:41,268 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting afc0c108c5f64a54b414656b4f5d241b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307679446 2023-04-12 13:54:41,268 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5a47ea6879284a9180281b69ba870147, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681307679386 2023-04-12 13:54:41,268 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ba993c300de14cf285f8f16a93af4b94, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681307681186 2023-04-12 13:54:41,268 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5a85179c8598419d8da9c404a0a9e0aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307679446 2023-04-12 13:54:41,269 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3bc52315936c4ef29634df7ceec2e491, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681307681186 2023-04-12 13:54:41,281 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:41,285 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#187 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:41,304 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:41,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:41,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:41,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:41,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:41,308 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:41,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:41,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:41,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:41,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:41,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:41,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:41,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:41,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:41,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:41,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:41,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:41,311 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:54:41,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:41,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:41,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:41,312 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:41,312 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:41,312 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:41,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:41,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:41,327 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/01a4bd723a104aa8be35370b4e6ef779 2023-04-12 13:54:41,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307741330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307741333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307741333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307741434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307741436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307741437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307741577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307741638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307741640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307741641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307741685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,698 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/cc621af8e2aa420f9b157bbbee47184d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/cc621af8e2aa420f9b157bbbee47184d 2023-04-12 13:54:41,703 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/eacb97d9a7e84e62b2da7759708f25c0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/eacb97d9a7e84e62b2da7759708f25c0 2023-04-12 13:54:41,704 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into cc621af8e2aa420f9b157bbbee47184d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:41,704 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:41,704 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307681265; duration=0sec 2023-04-12 13:54:41,704 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:41,704 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:41,709 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:41,709 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:41,709 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:41,709 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/c57600d9ef8d42c1a32eaf1ed8bca5e4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b82af56af245491eb05943713c60ae65, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/37fffa21ff0f43ffbf84d55e9eac0577] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=32.5 K 2023-04-12 13:54:41,709 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c57600d9ef8d42c1a32eaf1ed8bca5e4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681307679386 2023-04-12 13:54:41,710 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b82af56af245491eb05943713c60ae65, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307679446 2023-04-12 13:54:41,710 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 37fffa21ff0f43ffbf84d55e9eac0577, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681307681186 2023-04-12 13:54:41,714 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into eacb97d9a7e84e62b2da7759708f25c0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:41,714 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:41,714 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307681265; duration=0sec 2023-04-12 13:54:41,715 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:41,718 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:41,739 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/ca210053df8d4998b9abda8c55b3a48e 2023-04-12 13:54:41,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307741942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307741943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:41,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:41,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307741946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,137 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b5007f079f5f4f0eb68073a50ebf9414 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b5007f079f5f4f0eb68073a50ebf9414 2023-04-12 13:54:42,142 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into b5007f079f5f4f0eb68073a50ebf9414(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:42,142 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:42,143 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=13, startTime=1681307681265; duration=0sec 2023-04-12 13:54:42,143 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:42,151 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/28f09e16b95045b69e4ea4ef48210d01 2023-04-12 13:54:42,155 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/01a4bd723a104aa8be35370b4e6ef779 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/01a4bd723a104aa8be35370b4e6ef779 2023-04-12 13:54:42,159 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/01a4bd723a104aa8be35370b4e6ef779, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 13:54:42,160 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/ca210053df8d4998b9abda8c55b3a48e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ca210053df8d4998b9abda8c55b3a48e 2023-04-12 13:54:42,165 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ca210053df8d4998b9abda8c55b3a48e, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 13:54:42,176 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/28f09e16b95045b69e4ea4ef48210d01 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/28f09e16b95045b69e4ea4ef48210d01 2023-04-12 13:54:42,181 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/28f09e16b95045b69e4ea4ef48210d01, entries=150, sequenceid=82, filesize=11.6 K 2023-04-12 13:54:42,181 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 8e79b7aef32bdb05b28a7dce92e13c3a in 870ms, sequenceid=82, compaction requested=false 2023-04-12 13:54:42,181 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:42,181 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:42,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:42,183 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:42,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:42,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:42,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac43e2b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:42,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:42,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:42,188 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:42,188 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:42,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:42,189 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:42,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:42,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:42,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:42,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:42,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:42,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:42,192 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:42,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:42,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:42,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:42,193 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:42,194 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:42,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:42,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:42,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:42,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:42,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:42,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:42,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:42,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:42,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:42,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:42,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:42,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:42,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:42,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,199 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:42,199 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:42,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:42,299 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:42,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:42,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:42,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:42,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:42,302 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:42,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:42,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:42,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:42,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:42,336 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,336 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:42,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:42,336 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,336 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:42,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:42,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:42,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:42,338 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:42,338 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:42,338 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:54:42,338 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:42,339 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:42,339 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:42,339 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:42,339 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:42,339 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:42,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:42,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:42,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307742464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307742465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307742467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307742568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307742568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307742570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307742689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,753 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/e05982eb98724704a2a12566f73ffd40 2023-04-12 13:54:42,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307742772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307742772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:42,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307742773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:42,881 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 13:54:43,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307743075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307743076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307743077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,170 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/9a4396684d5f4515bff593628b21bd26 2023-04-12 13:54:43,190 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/0d69fdc9459244d0986a39ec87d5b38e 2023-04-12 13:54:43,196 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/e05982eb98724704a2a12566f73ffd40 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e05982eb98724704a2a12566f73ffd40 2023-04-12 13:54:43,204 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e05982eb98724704a2a12566f73ffd40, entries=150, sequenceid=98, filesize=11.6 K 2023-04-12 13:54:43,205 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/9a4396684d5f4515bff593628b21bd26 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9a4396684d5f4515bff593628b21bd26 2023-04-12 13:54:43,209 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9a4396684d5f4515bff593628b21bd26, entries=150, sequenceid=98, filesize=11.6 K 2023-04-12 13:54:43,213 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/0d69fdc9459244d0986a39ec87d5b38e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/0d69fdc9459244d0986a39ec87d5b38e 2023-04-12 13:54:43,217 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/0d69fdc9459244d0986a39ec87d5b38e, entries=150, sequenceid=98, filesize=11.6 K 2023-04-12 13:54:43,218 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8e79b7aef32bdb05b28a7dce92e13c3a in 880ms, sequenceid=98, compaction requested=true 2023-04-12 13:54:43,218 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:43,218 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:43,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:43,222 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:43,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd629f0[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:43,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:43,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,228 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:43,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:43,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:43,228 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,228 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:43,229 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,229 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:43,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:43,232 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:43,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:43,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:43,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:43,232 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:43,234 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,234 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:43,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,235 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,235 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,236 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,239 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,239 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,239 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:43,239 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:43,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:43,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:43,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:43,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:43,239 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:43,239 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:43,240 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,240 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:43,240 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:43,240 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,240 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,240 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:43,340 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:43,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:43,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:43,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:43,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:43,344 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:43,344 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:43,344 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:43,344 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:43,348 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,348 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,350 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:43,350 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:43,350 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,350 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:43,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:43,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:43,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:43,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:43,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:43,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:43,352 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:54:43,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:43,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:43,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:43,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:43,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:43,352 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:43,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:43,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:43,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307743589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307743589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307743590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307743591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307743694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307743694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307743694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307743695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,762 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/f09eb6323839446e83dcfdba6f6888d5 2023-04-12 13:54:43,774 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a 2023-04-12 13:54:43,785 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/f206eb15db8a4dc6a84045181973cfd7 2023-04-12 13:54:43,790 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/f09eb6323839446e83dcfdba6f6888d5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f09eb6323839446e83dcfdba6f6888d5 2023-04-12 13:54:43,795 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f09eb6323839446e83dcfdba6f6888d5, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:54:43,796 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a 2023-04-12 13:54:43,802 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:54:43,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/f206eb15db8a4dc6a84045181973cfd7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/f206eb15db8a4dc6a84045181973cfd7 2023-04-12 13:54:43,807 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/f206eb15db8a4dc6a84045181973cfd7, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:54:43,807 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8e79b7aef32bdb05b28a7dce92e13c3a in 455ms, sequenceid=117, compaction requested=true 2023-04-12 13:54:43,807 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:43,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:43,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:43,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd142c8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:43,813 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:43,813 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,814 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:43,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:43,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:43,815 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,815 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:43,817 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:43,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:43,820 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:43,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:43,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:43,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:43,820 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:43,821 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,821 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,821 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,822 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,823 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:43,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:43,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:43,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:43,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:43,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:43,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:43,826 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:43,826 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:43,826 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,826 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,826 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,826 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,826 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:43,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:43,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:54:43,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:43,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:43,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:43,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:43,900 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:43,900 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:43,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307743917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307743918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307743919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:43,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307743920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,926 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:43,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:43,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:43,929 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:43,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:43,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:43,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:43,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,930 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,930 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:43,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:43,931 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,931 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:43,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:43,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:43,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:43,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:43,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:43,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:43,932 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:43,935 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,935 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:43,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ac6292c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:43,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:43,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,938 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:43,938 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,938 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:43,939 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,939 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:43,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:43,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:43,942 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:43,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:43,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:43,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:43,942 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:43,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:43,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,944 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:43,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,945 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:43,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:43,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:43,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:43,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:43,948 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,948 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:43,948 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:43,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:43,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307744022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307744023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307744024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307744024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,048 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,052 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,056 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,056 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,056 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,056 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,058 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41587739[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,061 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,061 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,061 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,069 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,070 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,072 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,073 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,076 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,076 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,076 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,176 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,183 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8f5f4b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,190 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,194 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,195 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,197 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:44,201 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,201 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,201 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307744225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307744226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307744226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307744229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,301 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,305 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,306 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,308 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,308 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,309 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,309 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f 2023-04-12 13:54:44,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e898b70[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,313 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,313 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,319 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,319 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,319 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,320 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,320 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,322 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,322 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,322 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,323 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,325 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,327 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,327 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:44,329 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,329 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,429 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,432 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155d44c4[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,446 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,446 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,447 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,447 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,447 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,450 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,450 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,450 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,450 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,450 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,452 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,452 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,452 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,453 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,454 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,460 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,460 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,461 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,461 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,461 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,461 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307744528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307744530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307744530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307744532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,561 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,565 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,566 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,569 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,569 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,569 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,569 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,571 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,571 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17be18e9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,579 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,579 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,581 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,581 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,581 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,582 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,586 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,586 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,687 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,690 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,694 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,694 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,694 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,694 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:44,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307744696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,699 DEBUG [Thread-1422] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:44,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,701 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,701 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,702 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623e6852[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,704 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,705 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,708 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,708 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:44,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,715 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,715 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/86cbb0a491dc4fdfa7be191c76ee2c8e 2023-04-12 13:54:44,815 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,818 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,822 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,822 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,822 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,822 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,822 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,824 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60eead6e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,826 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,827 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,827 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,827 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,830 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,833 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,833 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,833 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,833 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,833 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,836 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,836 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,836 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,838 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,845 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,845 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,945 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:44,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:44,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:44,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:44,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:44,948 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:44,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:44,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:44,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,952 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,952 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,952 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,952 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:44,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:44,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:44,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:44,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:44,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:44,954 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:44,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:44,956 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,956 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,956 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,957 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:44,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6aaed3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:44,959 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:44,959 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:44,960 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,962 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:44,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:44,965 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:44,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:44,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:44,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:44,965 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:44,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:44,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:44,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:44,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:44,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:44,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:44,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:44,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:44,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:44,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:44,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:44,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:44,973 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:44,973 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:44,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:44,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:45,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:45,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307745030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:45,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307745034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:45,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307745036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307745037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,073 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:45,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:45,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:45,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:45,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:45,077 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:45,077 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:45,077 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:45,077 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:45,078 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,078 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,079 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,079 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:45,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:45,079 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,079 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:45,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:45,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:45,082 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,082 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:45,082 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:45,082 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,082 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:45,083 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:45,084 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,084 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f48487[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:45,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:45,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,088 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:45,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:45,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:45,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:45,089 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:45,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:45,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:45,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:45,092 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:45,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:45,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:45,092 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:45,092 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:45,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:45,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,096 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:45,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:45,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:45,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,100 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,100 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:45,100 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:45,100 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:45,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:45,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b44354102ca841dd90e503517c862ed1 2023-04-12 13:54:45,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f 2023-04-12 13:54:45,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f, entries=150, sequenceid=131, filesize=11.7 K 2023-04-12 13:54:45,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/86cbb0a491dc4fdfa7be191c76ee2c8e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/86cbb0a491dc4fdfa7be191c76ee2c8e 2023-04-12 13:54:45,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/86cbb0a491dc4fdfa7be191c76ee2c8e, entries=150, sequenceid=131, filesize=11.7 K 2023-04-12 13:54:45,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b44354102ca841dd90e503517c862ed1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b44354102ca841dd90e503517c862ed1 2023-04-12 13:54:45,200 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:45,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:45,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:45,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b44354102ca841dd90e503517c862ed1, entries=150, sequenceid=131, filesize=11.7 K 2023-04-12 13:54:45,206 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8e79b7aef32bdb05b28a7dce92e13c3a in 1307ms, sequenceid=131, compaction requested=true 2023-04-12 13:54:45,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:45,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:45,206 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:45,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:45,206 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:45,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:45,211 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:45,211 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:45,211 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,211 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/eacb97d9a7e84e62b2da7759708f25c0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/01a4bd723a104aa8be35370b4e6ef779, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e05982eb98724704a2a12566f73ffd40, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f09eb6323839446e83dcfdba6f6888d5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=58.2 K 2023-04-12 13:54:45,211 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:45,211 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:45,211 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,212 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/cc621af8e2aa420f9b157bbbee47184d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ca210053df8d4998b9abda8c55b3a48e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9a4396684d5f4515bff593628b21bd26, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/86cbb0a491dc4fdfa7be191c76ee2c8e] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=58.2 K 2023-04-12 13:54:45,212 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting eacb97d9a7e84e62b2da7759708f25c0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681307681186 2023-04-12 13:54:45,212 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting cc621af8e2aa420f9b157bbbee47184d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681307681186 2023-04-12 13:54:45,212 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ca210053df8d4998b9abda8c55b3a48e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681307681209 2023-04-12 13:54:45,212 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 01a4bd723a104aa8be35370b4e6ef779, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681307681209 2023-04-12 13:54:45,213 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9a4396684d5f4515bff593628b21bd26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681307681327 2023-04-12 13:54:45,213 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e05982eb98724704a2a12566f73ffd40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681307681327 2023-04-12 13:54:45,213 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c5c5953ec1124ff5b8e4ffb1d0f2f56a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307682456 2023-04-12 13:54:45,213 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f09eb6323839446e83dcfdba6f6888d5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307682456 2023-04-12 13:54:45,213 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 86cbb0a491dc4fdfa7be191c76ee2c8e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681307683588 2023-04-12 13:54:45,214 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bf43b7fd4f5e4e3b9d07a0f6ecb8f63f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681307683588 2023-04-12 13:54:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:45,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:45,220 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:45,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:45,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:45,220 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:45,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,223 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:45,223 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:45,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:45,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:45,225 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#201 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:45,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:45,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:45,244 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:45,244 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,244 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:45,244 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:45,244 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:45,245 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:45,246 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#202 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:45,675 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/0a37c33b7e844b528301a54bb00ddaa8 2023-04-12 13:54:45,680 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/bec12d3c8b5e4037ba7f3b20ad48617b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bec12d3c8b5e4037ba7f3b20ad48617b 2023-04-12 13:54:45,689 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into bec12d3c8b5e4037ba7f3b20ad48617b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:45,690 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:45,690 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=11, startTime=1681307685206; duration=0sec 2023-04-12 13:54:45,690 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:45,690 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:45,690 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/a378b443f0da4a1cbfcea8c74d6fee24 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a378b443f0da4a1cbfcea8c74d6fee24 2023-04-12 13:54:45,691 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/3e5b4c6f894143f8bc890986ee03f5bb 2023-04-12 13:54:45,692 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:45,692 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:45,692 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,693 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b5007f079f5f4f0eb68073a50ebf9414, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/28f09e16b95045b69e4ea4ef48210d01, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/0d69fdc9459244d0986a39ec87d5b38e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/f206eb15db8a4dc6a84045181973cfd7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b44354102ca841dd90e503517c862ed1] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=58.2 K 2023-04-12 13:54:45,693 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b5007f079f5f4f0eb68073a50ebf9414, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681307681186 2023-04-12 13:54:45,694 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 28f09e16b95045b69e4ea4ef48210d01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681307681209 2023-04-12 13:54:45,694 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 0d69fdc9459244d0986a39ec87d5b38e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681307681327 2023-04-12 13:54:45,695 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f206eb15db8a4dc6a84045181973cfd7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307682456 2023-04-12 13:54:45,695 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b44354102ca841dd90e503517c862ed1, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681307683588 2023-04-12 13:54:45,697 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into a378b443f0da4a1cbfcea8c74d6fee24(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:45,698 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:45,698 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=11, startTime=1681307685206; duration=0sec 2023-04-12 13:54:45,698 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:45,715 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b96e960dfe704b0bbaeaaf52d28ddd4e 2023-04-12 13:54:45,718 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#206 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:45,719 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/0a37c33b7e844b528301a54bb00ddaa8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0a37c33b7e844b528301a54bb00ddaa8 2023-04-12 13:54:45,724 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0a37c33b7e844b528301a54bb00ddaa8, entries=150, sequenceid=149, filesize=11.8 K 2023-04-12 13:54:45,726 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/3e5b4c6f894143f8bc890986ee03f5bb as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3e5b4c6f894143f8bc890986ee03f5bb 2023-04-12 13:54:45,733 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/378d61b1f76b4999b739523fdeaf4d50 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/378d61b1f76b4999b739523fdeaf4d50 2023-04-12 13:54:45,733 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3e5b4c6f894143f8bc890986ee03f5bb, entries=150, sequenceid=149, filesize=11.8 K 2023-04-12 13:54:45,735 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b96e960dfe704b0bbaeaaf52d28ddd4e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b96e960dfe704b0bbaeaaf52d28ddd4e 2023-04-12 13:54:45,739 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 378d61b1f76b4999b739523fdeaf4d50(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:45,739 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:45,739 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=11, startTime=1681307685206; duration=0sec 2023-04-12 13:54:45,739 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b96e960dfe704b0bbaeaaf52d28ddd4e, entries=150, sequenceid=149, filesize=11.8 K 2023-04-12 13:54:45,739 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:45,740 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 8e79b7aef32bdb05b28a7dce92e13c3a in 496ms, sequenceid=149, compaction requested=false 2023-04-12 13:54:45,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:45,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:45,744 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,745 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,745 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:45,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69991f33[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:45,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:45,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,748 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:45,748 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,748 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:45,750 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:45,753 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:45,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:45,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:45,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:45,754 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:45,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:45,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:45,757 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,759 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:45,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:45,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:45,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:45,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:45,763 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,763 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:45,763 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:45,762 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:45,763 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:45,763 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,763 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:45,863 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:45,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:45,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:45,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:45,867 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:45,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:45,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:45,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:45,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,867 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,876 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:45,876 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:45,876 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,876 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:45,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:45,878 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:45,886 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c895ab3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:45,888 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:45,888 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,890 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:45,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:45,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:45,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:45,896 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:45,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:45,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:45,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:45,896 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:45,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:45,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:45,899 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:45,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:45,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:45,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:45,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:45,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:45,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:45,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:45,904 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:45,905 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:45,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:45,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:45,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:45,905 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:45,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:45,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:45,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,005 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,009 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,011 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,013 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,013 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,013 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:46,013 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,015 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36df7fc7[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,017 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,017 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,018 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,020 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,023 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,023 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,027 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,028 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,031 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,032 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,032 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,032 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,032 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:46,059 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:54:46,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:46,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:46,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:46,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:46,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:46,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:46,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307746078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307746079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307686076, max timestamp=1681307686080Num uniques 859; 2023-04-12 13:54:46,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307686076, max timestamp=1681307686080Num uniques 859; 2023-04-12 13:54:46,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:54:46,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/d778939a11d04a00b7fa20ff906cf878 2023-04-12 13:54:46,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:54:46,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307686076, max timestamp=1681307686080Num uniques 859; 2023-04-12 13:54:46,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:54:46,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/b1dac72523f14d86a5a2c288f0cec1dd 2023-04-12 13:54:46,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307746125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307746126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,132 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,140 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,140 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,142 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,142 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:46,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,159 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a531527[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,164 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,164 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,164 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,166 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,170 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,170 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,182 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,182 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:46,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307746184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307746184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307746230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307746230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,282 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,286 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,287 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,288 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,290 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,290 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,290 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:46,290 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,291 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b12ffd9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,298 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,298 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,298 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,298 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,298 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,298 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,302 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,303 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,309 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,309 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307746388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307746389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,409 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,417 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:46,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d7868ca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,427 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,428 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,431 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,431 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307746434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307746434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:46,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,442 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,442 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,542 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,546 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/883cab695e9642449e11fc04a7a3a8e9 2023-04-12 13:54:46,548 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,551 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,551 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,551 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:46,551 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,553 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@309401e5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,557 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,557 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,559 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/d778939a11d04a00b7fa20ff906cf878 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/d778939a11d04a00b7fa20ff906cf878 2023-04-12 13:54:46,561 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,561 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,561 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/d778939a11d04a00b7fa20ff906cf878, entries=150, sequenceid=165, filesize=11.8 K 2023-04-12 13:54:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/b1dac72523f14d86a5a2c288f0cec1dd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b1dac72523f14d86a5a2c288f0cec1dd 2023-04-12 13:54:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,569 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,569 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b1dac72523f14d86a5a2c288f0cec1dd, entries=150, sequenceid=165, filesize=11.8 K 2023-04-12 13:54:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/883cab695e9642449e11fc04a7a3a8e9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/883cab695e9642449e11fc04a7a3a8e9 2023-04-12 13:54:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,573 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,577 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:46,577 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/883cab695e9642449e11fc04a7a3a8e9, entries=150, sequenceid=165, filesize=11.8 K 2023-04-12 13:54:46,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8e79b7aef32bdb05b28a7dce92e13c3a in 523ms, sequenceid=165, compaction requested=true 2023-04-12 13:54:46,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:46,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:46,583 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:46,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:46,583 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:46,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:46,584 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:46,584 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:46,584 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,584 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a378b443f0da4a1cbfcea8c74d6fee24, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0a37c33b7e844b528301a54bb00ddaa8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/d778939a11d04a00b7fa20ff906cf878] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=35.5 K 2023-04-12 13:54:46,584 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a378b443f0da4a1cbfcea8c74d6fee24, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681307683588 2023-04-12 13:54:46,584 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:46,585 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:46,585 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,585 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 0a37c33b7e844b528301a54bb00ddaa8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681307683916 2023-04-12 13:54:46,585 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bec12d3c8b5e4037ba7f3b20ad48617b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3e5b4c6f894143f8bc890986ee03f5bb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b1dac72523f14d86a5a2c288f0cec1dd] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=35.5 K 2023-04-12 13:54:46,585 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bec12d3c8b5e4037ba7f3b20ad48617b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681307683588 2023-04-12 13:54:46,585 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d778939a11d04a00b7fa20ff906cf878, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681307686055 2023-04-12 13:54:46,586 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3e5b4c6f894143f8bc890986ee03f5bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681307683916 2023-04-12 13:54:46,586 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b1dac72523f14d86a5a2c288f0cec1dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681307686055 2023-04-12 13:54:46,600 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:46,602 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#211 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:46,620 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/3f59febea83f4c438e45f1659f57ac87 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3f59febea83f4c438e45f1659f57ac87 2023-04-12 13:54:46,626 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 3f59febea83f4c438e45f1659f57ac87(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:46,627 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:46,627 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307686583; duration=0sec 2023-04-12 13:54:46,627 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:46,627 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:46,628 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:46,628 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:46,628 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,628 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/378d61b1f76b4999b739523fdeaf4d50, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b96e960dfe704b0bbaeaaf52d28ddd4e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/883cab695e9642449e11fc04a7a3a8e9] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=35.5 K 2023-04-12 13:54:46,629 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 378d61b1f76b4999b739523fdeaf4d50, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681307683588 2023-04-12 13:54:46,629 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b96e960dfe704b0bbaeaaf52d28ddd4e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681307683916 2023-04-12 13:54:46,629 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 883cab695e9642449e11fc04a7a3a8e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681307686055 2023-04-12 13:54:46,636 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#212 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:46,677 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,681 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,691 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,691 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,691 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-12 13:54:46,691 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:46,691 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:46,692 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:46,692 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:46,692 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:46,692 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:46,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:46,695 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:46,707 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/a9314260f8974f388fdcfea4f9ebea44 2023-04-12 13:54:46,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307746708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307746709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,727 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/e80ec15417894c22a56a5e1bdbd65506 2023-04-12 13:54:46,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307746737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307746738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,742 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=202 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/d6b52a739d4644a79cb0731a72ea8aa3 2023-04-12 13:54:46,748 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/a9314260f8974f388fdcfea4f9ebea44 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a9314260f8974f388fdcfea4f9ebea44 2023-04-12 13:54:46,752 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a9314260f8974f388fdcfea4f9ebea44, entries=150, sequenceid=202, filesize=11.8 K 2023-04-12 13:54:46,752 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/e80ec15417894c22a56a5e1bdbd65506 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e80ec15417894c22a56a5e1bdbd65506 2023-04-12 13:54:46,757 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e80ec15417894c22a56a5e1bdbd65506, entries=150, sequenceid=202, filesize=11.8 K 2023-04-12 13:54:46,758 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/d6b52a739d4644a79cb0731a72ea8aa3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d6b52a739d4644a79cb0731a72ea8aa3 2023-04-12 13:54:46,764 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d6b52a739d4644a79cb0731a72ea8aa3, entries=150, sequenceid=202, filesize=11.8 K 2023-04-12 13:54:46,765 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 8e79b7aef32bdb05b28a7dce92e13c3a in 74ms, sequenceid=202, compaction requested=false 2023-04-12 13:54:46,765 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:46,765 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,765 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429cc0fd[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,769 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,769 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,772 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,773 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,777 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,780 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,780 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,782 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,782 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,782 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,783 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,784 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,786 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,786 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,786 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,786 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:46,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,787 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,787 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,787 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:46,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:54:46,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:46,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:46,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:46,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:46,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:46,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:46,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307746871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307746872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,887 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:46,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:46,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:46,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:46,891 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,891 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,893 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,893 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:46,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:46,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:46,894 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,894 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:46,894 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:46,894 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:46,894 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,895 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:46,895 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,897 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,898 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eac1340[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:46,899 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:46,899 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,901 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:46,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:46,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:46,902 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:46,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:46,905 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:46,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:46,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:46,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:46,905 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:46,907 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:46,907 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,907 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:46,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,909 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:46,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:46,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:46,913 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:46,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,913 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:46,913 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:46,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:46,914 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:46,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:46,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307746974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:46,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:46,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307746974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,014 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,015 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/2cf856ae33894f3192221126fcf7fa74 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2cf856ae33894f3192221126fcf7fa74 2023-04-12 13:54:47,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,017 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,017 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,019 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,019 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,020 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,020 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,020 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 2cf856ae33894f3192221126fcf7fa74(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:47,020 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,020 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307686583; duration=0sec 2023-04-12 13:54:47,020 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:47,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,022 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,022 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,022 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,023 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,023 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202c0667[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,026 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,026 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,029 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,032 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,032 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,032 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,032 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,032 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,034 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,034 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,036 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,039 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,039 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,039 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,059 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/2411325824a349a8925b74db1dd28acf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2411325824a349a8925b74db1dd28acf 2023-04-12 13:54:47,071 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 2411325824a349a8925b74db1dd28acf(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:47,072 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,072 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=13, startTime=1681307686583; duration=0sec 2023-04-12 13:54:47,072 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:47,139 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,143 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,143 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,144 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,144 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,145 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,145 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,146 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,146 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,146 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,146 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,146 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a7b0ae[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,153 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,155 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,156 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,157 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,157 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,157 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,159 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,164 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,165 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:47,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,165 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307747177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41534 deadline: 1681307747177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/93f2463c6e4945cb8aaf3309cccbb34a 2023-04-12 13:54:47,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307747240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307747242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/e118cb1b3bb84ddf90f61bee46a984df 2023-04-12 13:54:47,265 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,270 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/d9e174319da84c43be5a34326183509c 2023-04-12 13:54:47,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/93f2463c6e4945cb8aaf3309cccbb34a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/93f2463c6e4945cb8aaf3309cccbb34a 2023-04-12 13:54:47,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,282 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,286 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/93f2463c6e4945cb8aaf3309cccbb34a, entries=150, sequenceid=215, filesize=11.8 K 2023-04-12 13:54:47,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,286 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/e118cb1b3bb84ddf90f61bee46a984df as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e118cb1b3bb84ddf90f61bee46a984df 2023-04-12 13:54:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1498056f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,289 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,289 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e118cb1b3bb84ddf90f61bee46a984df, entries=100, sequenceid=215, filesize=9.4 K 2023-04-12 13:54:47,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/d9e174319da84c43be5a34326183509c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d9e174319da84c43be5a34326183509c 2023-04-12 13:54:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,293 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,297 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,297 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,297 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,297 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,297 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,298 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,304 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,304 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d9e174319da84c43be5a34326183509c, entries=100, sequenceid=215, filesize=9.4 K 2023-04-12 13:54:47,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8e79b7aef32bdb05b28a7dce92e13c3a in 497ms, sequenceid=215, compaction requested=true 2023-04-12 13:54:47,311 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:47,311 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:47,312 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:47,311 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:47,312 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:47,319 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:47,319 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:47,319 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,320 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2cf856ae33894f3192221126fcf7fa74, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e80ec15417894c22a56a5e1bdbd65506, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e118cb1b3bb84ddf90f61bee46a984df] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=33.3 K 2023-04-12 13:54:47,320 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36480 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:47,320 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:47,320 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,320 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3f59febea83f4c438e45f1659f57ac87, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a9314260f8974f388fdcfea4f9ebea44, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/93f2463c6e4945cb8aaf3309cccbb34a] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=35.6 K 2023-04-12 13:54:47,320 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2cf856ae33894f3192221126fcf7fa74, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681307686055 2023-04-12 13:54:47,321 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3f59febea83f4c438e45f1659f57ac87, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681307686055 2023-04-12 13:54:47,321 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e80ec15417894c22a56a5e1bdbd65506, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1681307686113 2023-04-12 13:54:47,321 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a9314260f8974f388fdcfea4f9ebea44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1681307686113 2023-04-12 13:54:47,321 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e118cb1b3bb84ddf90f61bee46a984df, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681307686813 2023-04-12 13:54:47,321 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 93f2463c6e4945cb8aaf3309cccbb34a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681307686707 2023-04-12 13:54:47,333 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:47,335 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#220 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:47,357 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/8b7bb347844641f69f1efe0e2a2ad52c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/8b7bb347844641f69f1efe0e2a2ad52c 2023-04-12 13:54:47,364 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 8b7bb347844641f69f1efe0e2a2ad52c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:47,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,364 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307687311; duration=0sec 2023-04-12 13:54:47,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:47,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:47,365 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:47,366 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:47,366 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,366 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2411325824a349a8925b74db1dd28acf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d6b52a739d4644a79cb0731a72ea8aa3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d9e174319da84c43be5a34326183509c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=33.3 K 2023-04-12 13:54:47,366 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2411325824a349a8925b74db1dd28acf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1681307686055 2023-04-12 13:54:47,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d6b52a739d4644a79cb0731a72ea8aa3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=202, earliestPutTs=1681307686113 2023-04-12 13:54:47,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d9e174319da84c43be5a34326183509c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681307686813 2023-04-12 13:54:47,374 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:47,392 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/80bc785d74974730a5108671da4a06f3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/80bc785d74974730a5108671da4a06f3 2023-04-12 13:54:47,397 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 80bc785d74974730a5108671da4a06f3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:47,397 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,397 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=13, startTime=1681307687312; duration=0sec 2023-04-12 13:54:47,397 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:47,405 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,409 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,414 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,414 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:47,414 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:47,425 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/792e05da71d54335af40d27618704934 2023-04-12 13:54:47,444 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/ba2110052c4d4f5d813363d162fb67d3 2023-04-12 13:54:47,456 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/ad22268d7c6942baa9a1c4be1110695b 2023-04-12 13:54:47,461 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/792e05da71d54335af40d27618704934 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/792e05da71d54335af40d27618704934 2023-04-12 13:54:47,466 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/792e05da71d54335af40d27618704934, entries=150, sequenceid=238, filesize=11.8 K 2023-04-12 13:54:47,467 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/ba2110052c4d4f5d813363d162fb67d3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba2110052c4d4f5d813363d162fb67d3 2023-04-12 13:54:47,472 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba2110052c4d4f5d813363d162fb67d3, entries=150, sequenceid=238, filesize=11.8 K 2023-04-12 13:54:47,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/ad22268d7c6942baa9a1c4be1110695b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ad22268d7c6942baa9a1c4be1110695b 2023-04-12 13:54:47,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:47,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,491 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ad22268d7c6942baa9a1c4be1110695b, entries=150, sequenceid=238, filesize=11.8 K 2023-04-12 13:54:47,492 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=53.67 KB/54960 for 8e79b7aef32bdb05b28a7dce92e13c3a in 78ms, sequenceid=238, compaction requested=false 2023-04-12 13:54:47,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:47,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:54:47,493 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,493 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,493 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:47,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:47,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:47,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:47,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:47,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,499 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5032af[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,502 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,502 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,503 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,503 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,503 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,509 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,509 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,509 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,509 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,509 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,510 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,511 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,513 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,518 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,518 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,518 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,518 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,518 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,519 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,519 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,519 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,519 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,519 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,519 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,519 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307747536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41520 deadline: 1681307747536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/cecd7bfecdae49b89485d4ad1f8e4329 2023-04-12 13:54:47,619 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,623 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,623 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,623 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,623 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,623 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,623 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,625 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,625 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,625 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,625 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,626 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,626 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,626 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,627 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,627 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,627 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,627 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,627 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,629 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0f1ebc[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,630 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,631 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,632 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,632 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,632 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,635 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,635 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,638 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,638 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,639 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,639 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,639 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,642 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307747642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307747642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:47,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,647 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,647 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,648 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,748 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,752 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,752 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,752 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,752 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,753 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,753 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,754 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,755 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,757 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/be33e7443012488295618bfa2f2d85b3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/be33e7443012488295618bfa2f2d85b3 2023-04-12 13:54:47,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,758 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,760 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,761 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,761 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2720f993[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,763 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into be33e7443012488295618bfa2f2d85b3(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:47,763 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:47,763 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307687311; duration=0sec 2023-04-12 13:54:47,763 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:47,764 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,764 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,764 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,764 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,765 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,765 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,769 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,769 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,769 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,769 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,769 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,771 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,771 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,771 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,771 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,775 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,775 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,775 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:47,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307747845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307747846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,876 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:47,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:47,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:47,879 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:47,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:47,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:47,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,881 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,881 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:47,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:47,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:47,883 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,883 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:47,883 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:47,883 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:47,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:47,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:47,886 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:47,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c73365[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:47,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:47,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,896 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:47,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:47,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:47,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:47,901 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:47,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:47,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:47,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:47,901 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:47,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,904 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:47,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:47,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:47,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:47,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:47,909 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:47,909 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:47,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:47,909 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:47,909 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:47,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/b13f1c4d0503461395ac26dc3a593dcf 2023-04-12 13:54:47,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/67ece128c57f4db9b79ee37a1528fe7a 2023-04-12 13:54:47,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/cecd7bfecdae49b89485d4ad1f8e4329 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/cecd7bfecdae49b89485d4ad1f8e4329 2023-04-12 13:54:47,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/cecd7bfecdae49b89485d4ad1f8e4329, entries=250, sequenceid=250, filesize=16.4 K 2023-04-12 13:54:47,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/b13f1c4d0503461395ac26dc3a593dcf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b13f1c4d0503461395ac26dc3a593dcf 2023-04-12 13:54:47,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b13f1c4d0503461395ac26dc3a593dcf, entries=150, sequenceid=250, filesize=11.8 K 2023-04-12 13:54:47,999 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-12 13:54:48,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/67ece128c57f4db9b79ee37a1528fe7a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/67ece128c57f4db9b79ee37a1528fe7a 2023-04-12 13:54:48,004 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/67ece128c57f4db9b79ee37a1528fe7a, entries=150, sequenceid=250, filesize=11.8 K 2023-04-12 13:54:48,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8e79b7aef32bdb05b28a7dce92e13c3a in 513ms, sequenceid=250, compaction requested=true 2023-04-12 13:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:48,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:48,006 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:48,006 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:48,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:48,006 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:48,007 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:48,007 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:48,007 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:48,007 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:48,008 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:48,008 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:48,008 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/8b7bb347844641f69f1efe0e2a2ad52c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba2110052c4d4f5d813363d162fb67d3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b13f1c4d0503461395ac26dc3a593dcf] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=35.7 K 2023-04-12 13:54:48,008 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/be33e7443012488295618bfa2f2d85b3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/792e05da71d54335af40d27618704934, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/cecd7bfecdae49b89485d4ad1f8e4329] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=40.4 K 2023-04-12 13:54:48,008 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8b7bb347844641f69f1efe0e2a2ad52c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681307686120 2023-04-12 13:54:48,008 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting be33e7443012488295618bfa2f2d85b3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681307686120 2023-04-12 13:54:48,009 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ba2110052c4d4f5d813363d162fb67d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681307686861 2023-04-12 13:54:48,009 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:48,009 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 792e05da71d54335af40d27618704934, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681307686861 2023-04-12 13:54:48,009 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b13f1c4d0503461395ac26dc3a593dcf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681307687484 2023-04-12 13:54:48,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:48,010 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting cecd7bfecdae49b89485d4ad1f8e4329, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681307687484 2023-04-12 13:54:48,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:48,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:48,013 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:48,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:48,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:48,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:48,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,015 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:48,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:48,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,015 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:48,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:48,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:48,017 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:48,020 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:48,020 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:48,020 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:54:48,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:48,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:48,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:48,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:48,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:48,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:48,025 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:48,029 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:48,067 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/e0ca51d9d1eb48e8a7f9c4234e139420 2023-04-12 13:54:48,071 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/833910cbc7234c059233810ff437cd42 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/833910cbc7234c059233810ff437cd42 2023-04-12 13:54:48,071 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/3b87fa0b4e464f6f8f5290f0a4353bf5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3b87fa0b4e464f6f8f5290f0a4353bf5 2023-04-12 13:54:48,085 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 833910cbc7234c059233810ff437cd42(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:48,085 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:48,085 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307688005; duration=0sec 2023-04-12 13:54:48,085 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:48,086 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:48,086 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 3b87fa0b4e464f6f8f5290f0a4353bf5(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:48,086 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:48,086 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307688006; duration=0sec 2023-04-12 13:54:48,086 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:48,087 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:48,087 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:48,087 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:48,087 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/80bc785d74974730a5108671da4a06f3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ad22268d7c6942baa9a1c4be1110695b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/67ece128c57f4db9b79ee37a1528fe7a] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=35.7 K 2023-04-12 13:54:48,087 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 80bc785d74974730a5108671da4a06f3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681307686120 2023-04-12 13:54:48,088 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ad22268d7c6942baa9a1c4be1110695b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1681307686861 2023-04-12 13:54:48,088 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 67ece128c57f4db9b79ee37a1528fe7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681307687484 2023-04-12 13:54:48,098 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:48,115 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/4789bd92be614be18cd3602d7a7059e1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4789bd92be614be18cd3602d7a7059e1 2023-04-12 13:54:48,121 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 4789bd92be614be18cd3602d7a7059e1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:48,121 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:48,122 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=13, startTime=1681307688006; duration=0sec 2023-04-12 13:54:48,122 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:48,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:48,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:48,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307748160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307748162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307748246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307748250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307748263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307748265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307748466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307748468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,504 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/5b0853f0cf024b9c95199f73e3b3bff9 2023-04-12 13:54:48,538 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/284cd10347bc4d6db462509aad85ce66 2023-04-12 13:54:48,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/e0ca51d9d1eb48e8a7f9c4234e139420 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e0ca51d9d1eb48e8a7f9c4234e139420 2023-04-12 13:54:48,546 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e0ca51d9d1eb48e8a7f9c4234e139420, entries=150, sequenceid=271, filesize=11.9 K 2023-04-12 13:54:48,547 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/5b0853f0cf024b9c95199f73e3b3bff9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/5b0853f0cf024b9c95199f73e3b3bff9 2023-04-12 13:54:48,554 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/5b0853f0cf024b9c95199f73e3b3bff9, entries=150, sequenceid=271, filesize=11.9 K 2023-04-12 13:54:48,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/284cd10347bc4d6db462509aad85ce66 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/284cd10347bc4d6db462509aad85ce66 2023-04-12 13:54:48,562 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/284cd10347bc4d6db462509aad85ce66, entries=150, sequenceid=271, filesize=11.9 K 2023-04-12 13:54:48,563 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8e79b7aef32bdb05b28a7dce92e13c3a in 543ms, sequenceid=271, compaction requested=false 2023-04-12 13:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:48,563 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:48,563 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:48,563 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:48,563 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:48,564 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:48,565 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,566 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:48,566 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,566 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:48,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:48,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:48,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e9b2f8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:48,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:48,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,570 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:48,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:48,570 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:48,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:48,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:48,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:48,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:48,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:48,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:48,576 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:48,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:48,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:48,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:48,576 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:48,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:48,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:48,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,579 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,580 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:48,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:48,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:48,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:48,583 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,583 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:48,583 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:48,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:48,683 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:48,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:48,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:48,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:48,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:48,687 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:48,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:48,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:48,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:48,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:48,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:48,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:48,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:48,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:48,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:48,700 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:48,723 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/3309ebb0f34b47ad8b4636d87f894a5c 2023-04-12 13:54:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:48,727 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:48,736 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/2eac5ee3a8404fe08ba1576819f1cedd 2023-04-12 13:54:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307748773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307748773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307748774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307748876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307748876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:48,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:48,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307748876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307749079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307749079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307749080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,154 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/add08a5a0a3a414a8cf70e5abfca1f2d 2023-04-12 13:54:49,159 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/3309ebb0f34b47ad8b4636d87f894a5c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3309ebb0f34b47ad8b4636d87f894a5c 2023-04-12 13:54:49,163 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3309ebb0f34b47ad8b4636d87f894a5c, entries=150, sequenceid=286, filesize=11.9 K 2023-04-12 13:54:49,164 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/2eac5ee3a8404fe08ba1576819f1cedd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2eac5ee3a8404fe08ba1576819f1cedd 2023-04-12 13:54:49,172 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2eac5ee3a8404fe08ba1576819f1cedd, entries=150, sequenceid=286, filesize=11.9 K 2023-04-12 13:54:49,173 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/add08a5a0a3a414a8cf70e5abfca1f2d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/add08a5a0a3a414a8cf70e5abfca1f2d 2023-04-12 13:54:49,181 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/add08a5a0a3a414a8cf70e5abfca1f2d, entries=150, sequenceid=286, filesize=11.9 K 2023-04-12 13:54:49,182 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8e79b7aef32bdb05b28a7dce92e13c3a in 482ms, sequenceid=286, compaction requested=true 2023-04-12 13:54:49,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:49,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:49,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:49,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:49,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:49,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a41a5a7[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:49,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:49,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:49,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:49,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:49,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:49,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:49,196 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:49,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:49,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:49,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:49,196 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:49,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:49,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:49,199 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:49,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:49,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,205 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:49,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:49,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:49,205 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:49,206 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:49,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:49,206 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:49,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:49,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:49,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:49,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,306 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:49,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:49,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:49,309 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:49,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:49,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:49,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:49,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:49,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:49,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:49,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:49,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:49,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:49,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:49,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:49,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:49,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:49,313 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:49,313 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:54:49,314 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:49,314 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:49,314 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:49,314 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:49,314 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:49,314 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:49,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:49,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307749417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307749417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307749418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307749522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307749523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307749523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,725 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/c8749f586bc34ae4bd8cf13c3e32ca46 2023-04-12 13:54:49,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307749725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307749727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:49,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307749730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:49,746 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/639f2f7d57b74d9b9630621c072d4f4b 2023-04-12 13:54:50,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307750029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307750034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307750037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,156 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b26805cbf22f4603b0ad64790f7ba3ce 2023-04-12 13:54:50,161 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/c8749f586bc34ae4bd8cf13c3e32ca46 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8749f586bc34ae4bd8cf13c3e32ca46 2023-04-12 13:54:50,165 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8749f586bc34ae4bd8cf13c3e32ca46, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:54:50,165 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/639f2f7d57b74d9b9630621c072d4f4b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/639f2f7d57b74d9b9630621c072d4f4b 2023-04-12 13:54:50,169 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/639f2f7d57b74d9b9630621c072d4f4b, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:54:50,170 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/b26805cbf22f4603b0ad64790f7ba3ce as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b26805cbf22f4603b0ad64790f7ba3ce 2023-04-12 13:54:50,177 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b26805cbf22f4603b0ad64790f7ba3ce, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:54:50,178 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for 8e79b7aef32bdb05b28a7dce92e13c3a in 865ms, sequenceid=306, compaction requested=true 2023-04-12 13:54:50,178 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:50,178 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d311d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,189 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,189 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:50,195 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,196 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,196 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:50,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:54:50,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:50,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:50,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:50,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:50,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:50,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:50,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307750287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307750288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,296 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:50,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:50,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:50,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:50,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:50,299 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:50,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:50,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,302 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:50,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:50,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:50,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:50,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:50,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:50,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:50,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209b6b32[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,310 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,314 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,314 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,317 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,320 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,320 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,320 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:50,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,321 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307750390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307750391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,421 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:50,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:50,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:50,427 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:50,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:50,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:50,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,429 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:50,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:50,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:50,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:50,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:50,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:50,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@753073ab[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,436 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,440 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,440 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,446 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,446 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,446 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307750535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307750539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307750543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,546 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:50,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:50,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:50,550 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:50,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:50,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:50,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,553 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,553 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,553 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,553 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:50,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:50,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:50,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:50,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:50,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,556 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329b8c37[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,559 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,560 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,563 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,563 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,563 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,563 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,563 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,565 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,565 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,565 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,569 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,570 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307750593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307750595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,670 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:50,670 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/46dfacdf536c4f1f985744bd78c66f8c 2023-04-12 13:54:50,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:50,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:50,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:50,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:50,673 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:50,673 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,673 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:50,673 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:50,674 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,674 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,675 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:50,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:50,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:50,677 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,677 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:50,677 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:50,677 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311d46a0[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/e911344d3e864dcb94a26488066e16c2 2023-04-12 13:54:50,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,689 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,689 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,691 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,691 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,691 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,697 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,698 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,798 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:50,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:50,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:50,801 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:50,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:50,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:50,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,801 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,804 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,804 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,804 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,804 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:50,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:50,806 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,806 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:50,806 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:50,806 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,808 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,808 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa4794a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,819 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,819 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,821 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,822 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,824 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,824 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,824 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,824 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,824 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,824 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,825 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,825 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,825 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307750896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:50,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41512 deadline: 1681307750899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,925 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:50,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:50,929 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:50,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:50,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:50,930 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,930 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,932 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,932 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:50,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:50,933 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,933 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:50,934 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:50,934 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:50,940 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfff69e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:50,941 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:50,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,942 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:50,943 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:50,943 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,943 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:50,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,944 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:50,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:50,947 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:50,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:50,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:50,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:50,947 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:50,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:50,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:50,949 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,950 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,953 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,953 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,953 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:50,954 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:50,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:50,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:50,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:50,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:50,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:50,953 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:50,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:50,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:50,954 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:50,954 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:50,954 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:50,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,054 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:51,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:51,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:51,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:51,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:51,058 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:51,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:51,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:51,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:51,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:51,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:51,061 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,061 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:51,061 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:51,061 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:51,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,063 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:51,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526b2f02[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:51,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:51,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:51,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,066 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:51,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,068 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:51,070 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:51,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:51,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:51,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:51,071 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:51,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:51,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:51,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,077 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,077 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:51,077 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:51,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:51,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/d96a076c57244266b4960ee13aa633f8 2023-04-12 13:54:51,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/46dfacdf536c4f1f985744bd78c66f8c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46dfacdf536c4f1f985744bd78c66f8c 2023-04-12 13:54:51,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46dfacdf536c4f1f985744bd78c66f8c, entries=200, sequenceid=321, filesize=14.3 K 2023-04-12 13:54:51,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/e911344d3e864dcb94a26488066e16c2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e911344d3e864dcb94a26488066e16c2 2023-04-12 13:54:51,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e911344d3e864dcb94a26488066e16c2, entries=150, sequenceid=321, filesize=11.9 K 2023-04-12 13:54:51,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/d96a076c57244266b4960ee13aa633f8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d96a076c57244266b4960ee13aa633f8 2023-04-12 13:54:51,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d96a076c57244266b4960ee13aa633f8, entries=150, sequenceid=321, filesize=11.9 K 2023-04-12 13:54:51,122 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8e79b7aef32bdb05b28a7dce92e13c3a in 860ms, sequenceid=321, compaction requested=true 2023-04-12 13:54:51,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:51,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:51,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:51,123 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:51,123 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:51,123 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:51,125 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63800 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:51,125 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:51,125 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,125 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/833910cbc7234c059233810ff437cd42, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e0ca51d9d1eb48e8a7f9c4234e139420, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3309ebb0f34b47ad8b4636d87f894a5c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8749f586bc34ae4bd8cf13c3e32ca46, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46dfacdf536c4f1f985744bd78c66f8c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=62.3 K 2023-04-12 13:54:51,125 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61360 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:51,125 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:51,125 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 833910cbc7234c059233810ff437cd42, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681307687484 2023-04-12 13:54:51,125 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,125 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3b87fa0b4e464f6f8f5290f0a4353bf5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/5b0853f0cf024b9c95199f73e3b3bff9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2eac5ee3a8404fe08ba1576819f1cedd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/639f2f7d57b74d9b9630621c072d4f4b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e911344d3e864dcb94a26488066e16c2] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=59.9 K 2023-04-12 13:54:51,126 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e0ca51d9d1eb48e8a7f9c4234e139420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681307687524 2023-04-12 13:54:51,126 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3b87fa0b4e464f6f8f5290f0a4353bf5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681307687484 2023-04-12 13:54:51,126 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3309ebb0f34b47ad8b4636d87f894a5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681307688158 2023-04-12 13:54:51,126 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 5b0853f0cf024b9c95199f73e3b3bff9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681307687524 2023-04-12 13:54:51,127 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c8749f586bc34ae4bd8cf13c3e32ca46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681307688771 2023-04-12 13:54:51,127 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2eac5ee3a8404fe08ba1576819f1cedd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681307688158 2023-04-12 13:54:51,127 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 46dfacdf536c4f1f985744bd78c66f8c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681307689390 2023-04-12 13:54:51,127 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 639f2f7d57b74d9b9630621c072d4f4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681307688771 2023-04-12 13:54:51,128 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e911344d3e864dcb94a26488066e16c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681307689390 2023-04-12 13:54:51,137 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#243 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:51,139 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#244 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:51,152 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/6f70178ee9ca47249116d108f41aeecd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6f70178ee9ca47249116d108f41aeecd 2023-04-12 13:54:51,154 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/bb05fe081ef14913bf8a4cb8509fce45 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bb05fe081ef14913bf8a4cb8509fce45 2023-04-12 13:54:51,157 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 6f70178ee9ca47249116d108f41aeecd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:51,157 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:51,157 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=11, startTime=1681307691123; duration=0sec 2023-04-12 13:54:51,158 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:51,158 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:51,158 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into bb05fe081ef14913bf8a4cb8509fce45(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:51,158 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:51,158 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=11, startTime=1681307691123; duration=0sec 2023-04-12 13:54:51,158 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:51,159 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61360 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:51,159 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:51,159 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,159 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4789bd92be614be18cd3602d7a7059e1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/284cd10347bc4d6db462509aad85ce66, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/add08a5a0a3a414a8cf70e5abfca1f2d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b26805cbf22f4603b0ad64790f7ba3ce, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d96a076c57244266b4960ee13aa633f8] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=59.9 K 2023-04-12 13:54:51,160 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4789bd92be614be18cd3602d7a7059e1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681307687484 2023-04-12 13:54:51,160 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 284cd10347bc4d6db462509aad85ce66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681307687524 2023-04-12 13:54:51,160 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting add08a5a0a3a414a8cf70e5abfca1f2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681307688158 2023-04-12 13:54:51,161 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b26805cbf22f4603b0ad64790f7ba3ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681307688771 2023-04-12 13:54:51,161 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d96a076c57244266b4960ee13aa633f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681307689390 2023-04-12 13:54:51,170 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#245 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:51,177 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:51,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:51,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:51,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:51,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:51,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:51,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,182 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:51,184 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,184 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:51,185 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:54:51,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:51,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:51,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:51,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:51,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:51,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:51,194 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/2a03555c8b724529a3af2b9a34d69836 2023-04-12 13:54:51,209 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/9af4675aadd14ea9943e0b6f9c31e45d 2023-04-12 13:54:51,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:51,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:51,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307751423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307751425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307751526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307751528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307751546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307751549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307751551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,581 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/4ee6b08318da4dbdaf1cb3f4456713fd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4ee6b08318da4dbdaf1cb3f4456713fd 2023-04-12 13:54:51,586 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 4ee6b08318da4dbdaf1cb3f4456713fd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:51,586 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:51,586 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=11, startTime=1681307691123; duration=0sec 2023-04-12 13:54:51,587 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:51,626 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/55b256e8146f42f5804a5263a2d6fdb6 2023-04-12 13:54:51,630 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/2a03555c8b724529a3af2b9a34d69836 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2a03555c8b724529a3af2b9a34d69836 2023-04-12 13:54:51,634 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2a03555c8b724529a3af2b9a34d69836, entries=150, sequenceid=340, filesize=11.9 K 2023-04-12 13:54:51,635 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/9af4675aadd14ea9943e0b6f9c31e45d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9af4675aadd14ea9943e0b6f9c31e45d 2023-04-12 13:54:51,638 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9af4675aadd14ea9943e0b6f9c31e45d, entries=150, sequenceid=340, filesize=11.9 K 2023-04-12 13:54:51,639 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/55b256e8146f42f5804a5263a2d6fdb6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/55b256e8146f42f5804a5263a2d6fdb6 2023-04-12 13:54:51,642 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/55b256e8146f42f5804a5263a2d6fdb6, entries=150, sequenceid=340, filesize=11.9 K 2023-04-12 13:54:51,643 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8e79b7aef32bdb05b28a7dce92e13c3a in 458ms, sequenceid=340, compaction requested=false 2023-04-12 13:54:51,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:51,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:51,645 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,645 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9985ba9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:51,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:51,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,649 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:51,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,649 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:51,651 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,651 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:51,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:51,654 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:51,654 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:51,654 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:51,654 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:51,654 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:51,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,656 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,657 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,658 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:51,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:51,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,669 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,669 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,669 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:51,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:51,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-12 13:54:51,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:51,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:51,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:51,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:51,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:51,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:51,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307751767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,770 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:51,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:51,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307751769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:51,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:51,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:51,773 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:51,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:51,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:51,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,774 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,774 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,775 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,775 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:51,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:51,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:51,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:51,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:51,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:51,778 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@609f1df9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:51,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:51,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,781 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,781 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,781 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:51,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:51,784 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:51,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:51,786 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:51,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:51,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:51,787 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:51,787 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:51,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,789 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:51,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,793 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,793 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:51,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307751870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:51,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307751871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,893 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:51,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:51,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:51,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:51,897 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:51,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:51,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:51,898 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,898 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,899 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,899 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,899 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,900 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:51,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:51,901 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,901 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:51,901 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:51,901 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:51,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:51,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23975c72[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:51,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:51,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,907 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:51,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:51,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:51,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:51,909 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:51,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:51,912 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:51,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:51,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:51,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:51,912 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:51,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:51,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:51,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,916 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:51,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:51,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:51,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:51,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:51,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:51,920 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:51,920 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:52,020 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:52,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:52,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:52,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:52,023 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:52,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:52,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:52,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:52,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:52,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:52,026 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,026 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:52,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:52,027 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,027 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:52,027 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:52,027 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:52,029 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:52,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280f743d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:52,031 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:52,031 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,032 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:52,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:52,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:52,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:52,035 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:52,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:52,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:52,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:52,037 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:52,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:52,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:52,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:52,037 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:52,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,039 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:52,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:52,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:52,040 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,041 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:52,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,046 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:52,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:52,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:52,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,046 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:52,046 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:52,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:52,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307752072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307752075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,146 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:52,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:52,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:52,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:52,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:52,150 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:52,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:52,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:52,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:52,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:52,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:52,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:52,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:52,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:52,154 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,154 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:52,154 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:52,154 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:52,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:52,155 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63813786[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:52,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:52,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:52,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:52,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:52,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:52,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,159 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:52,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:52,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:52,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:52,162 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:52,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:52,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:52,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:52,162 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:52,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5bb71a6d5f3c47a991efa1ec82fdca26 2023-04-12 13:54:52,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:52,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:52,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:52,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:52,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,166 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:52,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:52,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:52,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:52,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:52,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:52,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:52,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:52,174 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:52,174 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:52,173 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:52,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:52,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/9c13964f71374f7d897aa5b3cddc570b 2023-04-12 13:54:52,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/adff3bded3da4c7d8ec328d36ca5943f 2023-04-12 13:54:52,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5bb71a6d5f3c47a991efa1ec82fdca26 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5bb71a6d5f3c47a991efa1ec82fdca26 2023-04-12 13:54:52,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5bb71a6d5f3c47a991efa1ec82fdca26, entries=150, sequenceid=358, filesize=11.9 K 2023-04-12 13:54:52,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/9c13964f71374f7d897aa5b3cddc570b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9c13964f71374f7d897aa5b3cddc570b 2023-04-12 13:54:52,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9c13964f71374f7d897aa5b3cddc570b, entries=150, sequenceid=358, filesize=11.9 K 2023-04-12 13:54:52,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/adff3bded3da4c7d8ec328d36ca5943f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/adff3bded3da4c7d8ec328d36ca5943f 2023-04-12 13:54:52,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/adff3bded3da4c7d8ec328d36ca5943f, entries=150, sequenceid=358, filesize=11.9 K 2023-04-12 13:54:52,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8e79b7aef32bdb05b28a7dce92e13c3a in 458ms, sequenceid=358, compaction requested=true 2023-04-12 13:54:52,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:52,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:52,205 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:52,205 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:52,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:52,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:52,207 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:52,208 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,208 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6f70178ee9ca47249116d108f41aeecd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2a03555c8b724529a3af2b9a34d69836, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5bb71a6d5f3c47a991efa1ec82fdca26] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=36.4 K 2023-04-12 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:52,208 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,208 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bb05fe081ef14913bf8a4cb8509fce45, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9af4675aadd14ea9943e0b6f9c31e45d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9c13964f71374f7d897aa5b3cddc570b] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=36.4 K 2023-04-12 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6f70178ee9ca47249116d108f41aeecd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681307689390 2023-04-12 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bb05fe081ef14913bf8a4cb8509fce45, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681307689390 2023-04-12 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2a03555c8b724529a3af2b9a34d69836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681307690282 2023-04-12 13:54:52,209 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9af4675aadd14ea9943e0b6f9c31e45d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681307690282 2023-04-12 13:54:52,209 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5bb71a6d5f3c47a991efa1ec82fdca26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681307691424 2023-04-12 13:54:52,209 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9c13964f71374f7d897aa5b3cddc570b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681307691424 2023-04-12 13:54:52,216 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#252 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:52,216 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:52,224 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/6a0997a071d64228b42fffc2dbed2eba as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6a0997a071d64228b42fffc2dbed2eba 2023-04-12 13:54:52,228 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 6a0997a071d64228b42fffc2dbed2eba(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:52,229 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:52,229 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307692205; duration=0sec 2023-04-12 13:54:52,229 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:52,229 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:52,229 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:52,230 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:52,230 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,230 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4ee6b08318da4dbdaf1cb3f4456713fd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/55b256e8146f42f5804a5263a2d6fdb6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/adff3bded3da4c7d8ec328d36ca5943f] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=36.4 K 2023-04-12 13:54:52,230 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4ee6b08318da4dbdaf1cb3f4456713fd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681307689390 2023-04-12 13:54:52,230 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 55b256e8146f42f5804a5263a2d6fdb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681307690282 2023-04-12 13:54:52,231 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting adff3bded3da4c7d8ec328d36ca5943f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681307691424 2023-04-12 13:54:52,236 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:52,274 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:52,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:52,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:52,277 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:52,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:52,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:52,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:52,278 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,278 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,279 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:52,279 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:52,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:52,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:52,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:52,280 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:52,281 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:52,281 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:52,293 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/46ae3b163fbe4c0ebbfec7af3ad5374e 2023-04-12 13:54:52,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:52,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:52,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307752398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307752400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307752501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307752503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,629 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/1255ace0a8c341b79d745c8324d8d6cd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1255ace0a8c341b79d745c8324d8d6cd 2023-04-12 13:54:52,633 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 1255ace0a8c341b79d745c8324d8d6cd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:52,633 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:52,633 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307692205; duration=0sec 2023-04-12 13:54:52,633 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:52,645 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/7cc4ebd9ac334bc4abb2832210422669 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/7cc4ebd9ac334bc4abb2832210422669 2023-04-12 13:54:52,659 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 7cc4ebd9ac334bc4abb2832210422669(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:52,659 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:52,659 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=13, startTime=1681307692205; duration=0sec 2023-04-12 13:54:52,659 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:52,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307752704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:52,705 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/04ee87dd51014bcf8676eb0275203349 2023-04-12 13:54:52,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:52,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307752708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307753007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307753012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,126 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/2f93a2dded3247bba9c37c04435d3a52 2023-04-12 13:54:53,133 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/46ae3b163fbe4c0ebbfec7af3ad5374e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46ae3b163fbe4c0ebbfec7af3ad5374e 2023-04-12 13:54:53,137 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46ae3b163fbe4c0ebbfec7af3ad5374e, entries=150, sequenceid=374, filesize=11.9 K 2023-04-12 13:54:53,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/04ee87dd51014bcf8676eb0275203349 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/04ee87dd51014bcf8676eb0275203349 2023-04-12 13:54:53,142 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/04ee87dd51014bcf8676eb0275203349, entries=150, sequenceid=374, filesize=11.9 K 2023-04-12 13:54:53,143 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/2f93a2dded3247bba9c37c04435d3a52 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2f93a2dded3247bba9c37c04435d3a52 2023-04-12 13:54:53,146 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2f93a2dded3247bba9c37c04435d3a52, entries=150, sequenceid=374, filesize=11.9 K 2023-04-12 13:54:53,147 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8e79b7aef32bdb05b28a7dce92e13c3a in 866ms, sequenceid=374, compaction requested=false 2023-04-12 13:54:53,147 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:53,147 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:53,149 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b699eda[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:53,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:53,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:53,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:53,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:53,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:53,155 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:53,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:53,158 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:53,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:53,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:53,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:53,158 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:53,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:53,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:53,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:53,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,161 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,164 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:53,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:53,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:53,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:53,164 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:53,164 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:53,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:53,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:53,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:53,264 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:53,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:53,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:53,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:53,268 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:53,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:53,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:53,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:53,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:53,270 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:53,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:53,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:53,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:53,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:53,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:53,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:53,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:53,272 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:53,273 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:53,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:53,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:53,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307753547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307753549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307753558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,561 DEBUG [Thread-1426] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:53,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307753565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,568 DEBUG [Thread-1422] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:53,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307753572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,574 DEBUG [Thread-1424] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:53,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307753650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41536 deadline: 1681307753652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,681 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/0ad540149030456bb99c2f50b1d0e753 2023-04-12 13:54:53,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307753852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:53,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307753856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,091 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/929ad4e91bbc4ea7b603cc432911f0d4 2023-04-12 13:54:54,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307754155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307754159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,504 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/ea7b5220637947349bc49ed789b8f307 2023-04-12 13:54:54,508 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/0ad540149030456bb99c2f50b1d0e753 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0ad540149030456bb99c2f50b1d0e753 2023-04-12 13:54:54,513 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0ad540149030456bb99c2f50b1d0e753, entries=150, sequenceid=393, filesize=11.9 K 2023-04-12 13:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/929ad4e91bbc4ea7b603cc432911f0d4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/929ad4e91bbc4ea7b603cc432911f0d4 2023-04-12 13:54:54,517 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/929ad4e91bbc4ea7b603cc432911f0d4, entries=150, sequenceid=393, filesize=11.9 K 2023-04-12 13:54:54,518 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/ea7b5220637947349bc49ed789b8f307 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ea7b5220637947349bc49ed789b8f307 2023-04-12 13:54:54,522 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ea7b5220637947349bc49ed789b8f307, entries=150, sequenceid=393, filesize=11.9 K 2023-04-12 13:54:54,523 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8e79b7aef32bdb05b28a7dce92e13c3a in 1251ms, sequenceid=393, compaction requested=true 2023-04-12 13:54:54,523 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:54,523 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:54,526 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,527 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62843909[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:54,528 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:54,528 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,530 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:54,530 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:54,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:54,531 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,531 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:54,533 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:54,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:54,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:54,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:54,533 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:54,535 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,535 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,535 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,536 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,540 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,541 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:54,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,541 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:54,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,641 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:54,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:54,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:54,644 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:54,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:54,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:54,644 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:54,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,648 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:54,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:54,649 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:54,649 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:54,650 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:54,650 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:54,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:54,659 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/e5a0797800e645c388cac10434c89331 2023-04-12 13:54:54,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:54,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:54,670 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/be429db5a4e6436684de5683685147ca 2023-04-12 13:54:54,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307754686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,691 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/9656b85379124e3dafa17373cd34ec01 2023-04-12 13:54:54,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307754690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,698 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/e5a0797800e645c388cac10434c89331 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e5a0797800e645c388cac10434c89331 2023-04-12 13:54:54,702 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e5a0797800e645c388cac10434c89331, entries=150, sequenceid=408, filesize=11.9 K 2023-04-12 13:54:54,703 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/be429db5a4e6436684de5683685147ca as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/be429db5a4e6436684de5683685147ca 2023-04-12 13:54:54,707 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/be429db5a4e6436684de5683685147ca, entries=150, sequenceid=408, filesize=11.9 K 2023-04-12 13:54:54,708 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/9656b85379124e3dafa17373cd34ec01 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9656b85379124e3dafa17373cd34ec01 2023-04-12 13:54:54,712 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9656b85379124e3dafa17373cd34ec01, entries=150, sequenceid=408, filesize=11.9 K 2023-04-12 13:54:54,712 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8e79b7aef32bdb05b28a7dce92e13c3a in 62ms, sequenceid=408, compaction requested=true 2023-04-12 13:54:54,713 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:54,713 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:54,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:54,719 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:54,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f907f8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:54,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:54,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,724 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:54,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:54,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:54,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:54,726 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,726 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:54,726 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,726 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:54,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:54,729 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:54,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:54,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:54,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:54,729 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:54,731 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,731 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,731 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,731 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,732 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,735 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:54,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:54,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,735 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:54,735 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,735 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:54,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:54,793 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:54:54,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:54,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:54,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:54,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:54,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:54,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:54,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307754824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307754827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,835 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:54,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:54,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:54,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:54,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:54,839 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:54,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:54,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:54,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:54,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:54,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:54,843 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,843 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:54,843 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:54,843 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:54,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:54,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bae2b46[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:54,847 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:54,847 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:54,848 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,848 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:54,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:54,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:54,853 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:54,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:54,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:54,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:54,853 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:54,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,857 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,860 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:54,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:54,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:54,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,860 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:54,860 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:54,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307754928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:54,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307754930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,960 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:54,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:54,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:54,964 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:54,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:54,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:54,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:54,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,964 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,966 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:54,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:54,967 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:54,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:54,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:54,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:54,969 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,969 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c86e1a7[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:54,971 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:54,971 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,973 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:54,973 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:54,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:54,973 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:54,974 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:54,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:54,977 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:54,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:54,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:54,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:54,977 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:54,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:54,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:54,979 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,983 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:54,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:54,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:54,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:54,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:54,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:54,984 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:54,984 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:54,984 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:54,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,084 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,088 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,091 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,091 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,091 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,091 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:55,093 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,093 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,103 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,103 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f89d14f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,105 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,106 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,106 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,106 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,108 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,110 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,111 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,112 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,112 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,112 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,116 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,116 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,116 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,117 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,117 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,117 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307755131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307755133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/de27cb59108b4a54ae030b2ff8facc96 2023-04-12 13:54:55,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/6e032311857f4f60a59c25616896c038 2023-04-12 13:54:55,217 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/5a29d83023f74bcc823b24e22bd7d115 2023-04-12 13:54:55,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,227 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,229 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,229 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,230 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,230 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/de27cb59108b4a54ae030b2ff8facc96 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/de27cb59108b4a54ae030b2ff8facc96 2023-04-12 13:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:55,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,235 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/de27cb59108b4a54ae030b2ff8facc96, entries=200, sequenceid=426, filesize=14.3 K 2023-04-12 13:54:55,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/6e032311857f4f60a59c25616896c038 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6e032311857f4f60a59c25616896c038 2023-04-12 13:54:55,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6e032311857f4f60a59c25616896c038, entries=150, sequenceid=426, filesize=11.9 K 2023-04-12 13:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/5a29d83023f74bcc823b24e22bd7d115 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/5a29d83023f74bcc823b24e22bd7d115 2023-04-12 13:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5636d581[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,240 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,240 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,241 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,241 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,242 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,243 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/5a29d83023f74bcc823b24e22bd7d115, entries=150, sequenceid=426, filesize=11.9 K 2023-04-12 13:54:55,243 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8e79b7aef32bdb05b28a7dce92e13c3a in 451ms, sequenceid=426, compaction requested=true 2023-04-12 13:54:55,244 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,244 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:55,244 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:55,244 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,244 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-12 13:54:55,244 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,246 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64222 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:55,246 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,246 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,246 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,246 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,246 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,246 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-12 13:54:55,246 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:55,246 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:55,246 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,246 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,246 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6a0997a071d64228b42fffc2dbed2eba, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46ae3b163fbe4c0ebbfec7af3ad5374e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0ad540149030456bb99c2f50b1d0e753, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e5a0797800e645c388cac10434c89331, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/de27cb59108b4a54ae030b2ff8facc96] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=62.7 K 2023-04-12 13:54:55,246 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1255ace0a8c341b79d745c8324d8d6cd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/04ee87dd51014bcf8676eb0275203349, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/929ad4e91bbc4ea7b603cc432911f0d4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/be429db5a4e6436684de5683685147ca, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6e032311857f4f60a59c25616896c038] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=60.3 K 2023-04-12 13:54:55,247 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6a0997a071d64228b42fffc2dbed2eba, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681307691424 2023-04-12 13:54:55,247 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1255ace0a8c341b79d745c8324d8d6cd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681307691424 2023-04-12 13:54:55,247 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 46ae3b163fbe4c0ebbfec7af3ad5374e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681307691763 2023-04-12 13:54:55,247 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 04ee87dd51014bcf8676eb0275203349, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681307691763 2023-04-12 13:54:55,247 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 0ad540149030456bb99c2f50b1d0e753, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681307692387 2023-04-12 13:54:55,248 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 929ad4e91bbc4ea7b603cc432911f0d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681307692387 2023-04-12 13:54:55,248 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e5a0797800e645c388cac10434c89331, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681307693541 2023-04-12 13:54:55,248 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,248 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,248 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting be429db5a4e6436684de5683685147ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681307693541 2023-04-12 13:54:55,248 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting de27cb59108b4a54ae030b2ff8facc96, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681307694675 2023-04-12 13:54:55,248 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,249 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6e032311857f4f60a59c25616896c038, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681307694675 2023-04-12 13:54:55,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,249 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,251 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,255 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,256 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,256 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,260 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#267 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:55,260 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#268 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:55,356 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,359 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,361 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,362 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,363 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,363 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:55,363 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:55,364 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:55,371 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/91700dba6f1648c0a9d68c71253a791c 2023-04-12 13:54:55,380 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/d3dd9dd4b279407fb01af858b66a69a5 2023-04-12 13:54:55,389 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/27ac97b1692a4c6ea4944f6cfaadd037 2023-04-12 13:54:55,392 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/91700dba6f1648c0a9d68c71253a791c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/91700dba6f1648c0a9d68c71253a791c 2023-04-12 13:54:55,396 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/91700dba6f1648c0a9d68c71253a791c, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 13:54:55,397 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/d3dd9dd4b279407fb01af858b66a69a5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/d3dd9dd4b279407fb01af858b66a69a5 2023-04-12 13:54:55,400 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/d3dd9dd4b279407fb01af858b66a69a5, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 13:54:55,400 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/27ac97b1692a4c6ea4944f6cfaadd037 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/27ac97b1692a4c6ea4944f6cfaadd037 2023-04-12 13:54:55,404 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/27ac97b1692a4c6ea4944f6cfaadd037, entries=150, sequenceid=440, filesize=11.9 K 2023-04-12 13:54:55,405 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 8e79b7aef32bdb05b28a7dce92e13c3a in 42ms, sequenceid=440, compaction requested=true 2023-04-12 13:54:55,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:55,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,405 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,406 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,407 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,407 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c68301a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,409 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,410 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,410 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,410 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,412 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,415 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,415 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,416 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,416 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,421 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,422 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,422 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:55,445 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:54:55,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:55,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:55,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:55,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:55,445 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:55,445 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:55,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307695473, max timestamp=1681307695489Num uniques 859; 2023-04-12 13:54:55,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:55,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307695473, max timestamp=1681307695489Num uniques 859; 2023-04-12 13:54:55,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:55,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307695473, max timestamp=1681307695489Num uniques 859; 2023-04-12 13:54:55,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:55,522 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,525 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,525 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,527 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,527 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,527 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,527 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,529 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,529 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,529 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:55,529 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,529 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,531 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,532 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27b87de9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,534 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,534 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,534 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,534 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,541 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,543 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,543 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,545 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,545 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,545 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307755541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307755541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,549 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,549 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,549 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,649 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307755648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307755648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,653 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,654 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,654 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,655 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,655 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,656 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,656 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,656 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:55,656 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,658 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,658 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,658 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,658 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dac3f8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,659 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,660 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,664 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,664 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,664 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,670 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,670 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,671 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/8017a9c48a324c8694993e1abf881aa2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/8017a9c48a324c8694993e1abf881aa2 2023-04-12 13:54:55,672 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/005eab84b6c5443bb030609a8514e487 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/005eab84b6c5443bb030609a8514e487 2023-04-12 13:54:55,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,673 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,677 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 8017a9c48a324c8694993e1abf881aa2(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:55,677 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:55,677 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=11, startTime=1681307695244; duration=0sec 2023-04-12 13:54:55,677 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:55,678 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:54:55,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,679 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,679 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,680 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 005eab84b6c5443bb030609a8514e487(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:55,680 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:55,680 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=11, startTime=1681307695244; duration=0sec 2023-04-12 13:54:55,680 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:55,681 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73968 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:54:55,681 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:55,681 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,681 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/7cc4ebd9ac334bc4abb2832210422669, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2f93a2dded3247bba9c37c04435d3a52, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ea7b5220637947349bc49ed789b8f307, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9656b85379124e3dafa17373cd34ec01, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/5a29d83023f74bcc823b24e22bd7d115, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/27ac97b1692a4c6ea4944f6cfaadd037] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=72.2 K 2023-04-12 13:54:55,681 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 7cc4ebd9ac334bc4abb2832210422669, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681307691424 2023-04-12 13:54:55,682 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2f93a2dded3247bba9c37c04435d3a52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681307691763 2023-04-12 13:54:55,682 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ea7b5220637947349bc49ed789b8f307, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681307692387 2023-04-12 13:54:55,682 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 9656b85379124e3dafa17373cd34ec01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681307693541 2023-04-12 13:54:55,683 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5a29d83023f74bcc823b24e22bd7d115, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681307694675 2023-04-12 13:54:55,686 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 27ac97b1692a4c6ea4944f6cfaadd037, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681307694800 2023-04-12 13:54:55,696 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:55,779 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,782 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,782 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,782 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,782 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,782 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,783 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,785 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,785 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,785 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,785 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,787 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,787 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,787 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:55,787 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,789 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a09c7f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,791 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,792 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,792 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,802 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,802 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,806 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,810 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,810 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,810 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,810 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,810 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,810 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/2648689433634c32bf19abc17611f5c7 2023-04-12 13:54:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307755852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307755852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,911 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:55,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:55,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:55,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:55,914 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,916 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,916 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,916 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,916 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:55,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:55,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:55,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:55,918 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,918 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:55,918 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:55,918 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:55,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:55,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf59051[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:55,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:55,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,924 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:55,924 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,925 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:55,926 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:55,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:55,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:55,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:55,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:55,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:55,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:55,941 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,941 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,941 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:55,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,944 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:55,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:55,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:55,947 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:55,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:55,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:55,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:55,947 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:55,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:55,948 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:55,948 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:55,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:56,048 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:56,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:56,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:56,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:56,051 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:56,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:56,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:56,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:56,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,053 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:56,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:56,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:56,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:56,055 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:56,057 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc6e1f3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:56,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:56,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,060 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:56,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:56,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:56,064 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:56,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:56,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:56,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:56,064 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:56,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:56,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:56,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:56,067 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:56,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:56,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:56,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:56,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:56,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:56,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:56,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,071 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:56,071 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:56,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:56,105 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/4d2dcd2dbefa484b91ad518a4c05f4da as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4d2dcd2dbefa484b91ad518a4c05f4da 2023-04-12 13:54:56,109 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 4d2dcd2dbefa484b91ad518a4c05f4da(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:56,109 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:56,110 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=10, startTime=1681307695244; duration=0sec 2023-04-12 13:54:56,110 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:56,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307756156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307756157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,171 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:56,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:56,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:56,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:56,174 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:56,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:56,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:56,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:56,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:56,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:56,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,178 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:56,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:56,178 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,178 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:56,179 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:56,179 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:56,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1338f0[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:56,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:56,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:56,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:56,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:56,188 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:56,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:56,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:56,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:56,188 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:56,190 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,190 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:56,190 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:56,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:56,191 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:56,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:56,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:56,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:56,195 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:56,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,196 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:56,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/bc4278f989b24538aec54d24f92253ea 2023-04-12 13:54:56,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/21dec43afcca4ee6a1bebcdb89b50113 2023-04-12 13:54:56,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/2648689433634c32bf19abc17611f5c7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2648689433634c32bf19abc17611f5c7 2023-04-12 13:54:56,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2648689433634c32bf19abc17611f5c7, entries=200, sequenceid=450, filesize=14.3 K 2023-04-12 13:54:56,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/bc4278f989b24538aec54d24f92253ea as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bc4278f989b24538aec54d24f92253ea 2023-04-12 13:54:56,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bc4278f989b24538aec54d24f92253ea, entries=150, sequenceid=450, filesize=11.9 K 2023-04-12 13:54:56,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/21dec43afcca4ee6a1bebcdb89b50113 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/21dec43afcca4ee6a1bebcdb89b50113 2023-04-12 13:54:56,296 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:56,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:56,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/21dec43afcca4ee6a1bebcdb89b50113, entries=150, sequenceid=450, filesize=11.9 K 2023-04-12 13:54:56,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8e79b7aef32bdb05b28a7dce92e13c3a in 854ms, sequenceid=450, compaction requested=true 2023-04-12 13:54:56,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:56,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:56,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:56,299 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:56,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:56,299 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:56,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:56,299 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:56,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:56,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:56,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:56,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:56,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,300 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:56,300 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:56,301 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,301 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:56,301 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/005eab84b6c5443bb030609a8514e487, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/d3dd9dd4b279407fb01af858b66a69a5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bc4278f989b24538aec54d24f92253ea] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=36.7 K 2023-04-12 13:54:56,301 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:56,301 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,301 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/8017a9c48a324c8694993e1abf881aa2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/91700dba6f1648c0a9d68c71253a791c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2648689433634c32bf19abc17611f5c7] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=39.1 K 2023-04-12 13:54:56,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:56,301 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 005eab84b6c5443bb030609a8514e487, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681307694675 2023-04-12 13:54:56,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:56,302 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 8017a9c48a324c8694993e1abf881aa2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681307694675 2023-04-12 13:54:56,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,302 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d3dd9dd4b279407fb01af858b66a69a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681307694800 2023-04-12 13:54:56,302 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 91700dba6f1648c0a9d68c71253a791c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681307694800 2023-04-12 13:54:56,302 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bc4278f989b24538aec54d24f92253ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681307695438 2023-04-12 13:54:56,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:56,303 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2648689433634c32bf19abc17611f5c7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681307695438 2023-04-12 13:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:56,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:56,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:56,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:56,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:56,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:56,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:56,304 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-12 13:54:56,304 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:56,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:56,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:56,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:56,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:56,305 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:56,310 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:56,311 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#278 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:56,338 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/5755d14f782e4be18e9fd9782144f85f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5755d14f782e4be18e9fd9782144f85f 2023-04-12 13:54:56,343 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 5755d14f782e4be18e9fd9782144f85f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:56,343 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:56,343 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307696299; duration=0sec 2023-04-12 13:54:56,343 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:56,343 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-12 13:54:56,343 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-12 13:54:56,344 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-12 13:54:56,344 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. because compaction request was cancelled 2023-04-12 13:54:56,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:56,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:56,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307756673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307756673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,730 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/27669ab654be4b4f8518a0b056450a58 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/27669ab654be4b4f8518a0b056450a58 2023-04-12 13:54:56,733 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/9b6970db091a47c694ead5ee5d08f7c5 2023-04-12 13:54:56,734 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 27669ab654be4b4f8518a0b056450a58(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:56,734 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:56,734 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307696299; duration=0sec 2023-04-12 13:54:56,734 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:56,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307756778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307756778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307756980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:56,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307756980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,144 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/44995ba99af749d4afc4f1880df3db7c 2023-04-12 13:54:57,155 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/9770298171fa411290ef4ce9676e913e 2023-04-12 13:54:57,159 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/9b6970db091a47c694ead5ee5d08f7c5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9b6970db091a47c694ead5ee5d08f7c5 2023-04-12 13:54:57,163 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9b6970db091a47c694ead5ee5d08f7c5, entries=150, sequenceid=490, filesize=11.9 K 2023-04-12 13:54:57,163 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/44995ba99af749d4afc4f1880df3db7c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/44995ba99af749d4afc4f1880df3db7c 2023-04-12 13:54:57,167 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/44995ba99af749d4afc4f1880df3db7c, entries=150, sequenceid=490, filesize=11.9 K 2023-04-12 13:54:57,168 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/9770298171fa411290ef4ce9676e913e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9770298171fa411290ef4ce9676e913e 2023-04-12 13:54:57,172 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9770298171fa411290ef4ce9676e913e, entries=150, sequenceid=490, filesize=11.9 K 2023-04-12 13:54:57,172 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 8e79b7aef32bdb05b28a7dce92e13c3a in 868ms, sequenceid=490, compaction requested=true 2023-04-12 13:54:57,173 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:57,173 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:57,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:57,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,174 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,174 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,174 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:57,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d720209[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:57,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:57,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,178 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:57,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:57,178 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:57,180 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,180 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:57,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:57,183 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:57,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:57,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:57,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:57,183 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:57,184 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,184 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,184 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,187 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:57,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:57,192 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,192 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:57,192 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:57,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,193 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,193 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:57,284 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:54:57,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:57,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:57,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:57,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:57,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:57,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:57,293 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:57,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:57,295 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/f77fd2891e354952adb1369c7a84cbc3 2023-04-12 13:54:57,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:57,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:57,296 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:57,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:57,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:57,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,297 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:57,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:57,302 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,302 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:57,302 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:57,302 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:57,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3274ee[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:57,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:57,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/1e89ac65fdea4fbf83a5b4277e723563 2023-04-12 13:54:57,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:57,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:57,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307697304, max timestamp=1681307697311Num uniques 859; 2023-04-12 13:54:57,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307697304, max timestamp=1681307697311Num uniques 859; 2023-04-12 13:54:57,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307697304, max timestamp=1681307697311Num uniques 859; 2023-04-12 13:54:57,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:57,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:57,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:57,315 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:57,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:57,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:57,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:57,315 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:57,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,320 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:57,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:57,323 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,323 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:57,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,323 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:57,323 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307757338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307757339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,424 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:57,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:57,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:57,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:57,427 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:57,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:57,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:57,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,429 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:57,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:57,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3233db22[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:57,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:57,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:57,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:57,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:57,441 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:57,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:57,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:57,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:57,441 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:57,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,443 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,443 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307757441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,443 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,445 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307757443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,447 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:57,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,448 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:57,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:57,448 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:57,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,548 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:57,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:57,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:57,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:57,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:57,551 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:57,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:57,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:57,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:57,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:57,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:57,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:57,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:57,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:57,557 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4491b12c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:57,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:57,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:57,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:57,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:57,564 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:57,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:57,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:57,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:57,565 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:57,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41534 deadline: 1681307757569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,571 DEBUG [Thread-1426] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,573 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41548 deadline: 1681307757571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:57,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:57,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,575 DEBUG [Thread-1422] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:57,575 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:57,575 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:57,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41520 deadline: 1681307757607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,610 DEBUG [Thread-1424] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:54:57,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307757645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307757646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,675 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:57,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:57,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:57,679 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:57,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:57,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:57,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,684 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:57,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:57,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:57,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:57,685 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,685 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:57,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:57,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:57,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:57,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ae91fd[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:57,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:57,690 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,691 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:57,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:57,691 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:57,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,693 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:57,696 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:57,696 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:57,696 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:57,696 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:57,696 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:57,697 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:57,698 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:57,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:57,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:57,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:57,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:57,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:57,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:57,703 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:57,703 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:57,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:54:57,719 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/92325d2bcc9d46afb6e394dd4f76d535 2023-04-12 13:54:57,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/f77fd2891e354952adb1369c7a84cbc3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f77fd2891e354952adb1369c7a84cbc3 2023-04-12 13:54:57,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f77fd2891e354952adb1369c7a84cbc3, entries=150, sequenceid=502, filesize=11.9 K 2023-04-12 13:54:57,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/1e89ac65fdea4fbf83a5b4277e723563 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1e89ac65fdea4fbf83a5b4277e723563 2023-04-12 13:54:57,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1e89ac65fdea4fbf83a5b4277e723563, entries=150, sequenceid=502, filesize=11.9 K 2023-04-12 13:54:57,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/92325d2bcc9d46afb6e394dd4f76d535 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/92325d2bcc9d46afb6e394dd4f76d535 2023-04-12 13:54:57,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/92325d2bcc9d46afb6e394dd4f76d535, entries=150, sequenceid=502, filesize=11.9 K 2023-04-12 13:54:57,737 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 8e79b7aef32bdb05b28a7dce92e13c3a in 453ms, sequenceid=502, compaction requested=true 2023-04-12 13:54:57,737 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:57,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:57,737 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:57,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:57,737 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 13:54:57,738 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:57,738 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:54:57,739 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,739 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5755d14f782e4be18e9fd9782144f85f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9b6970db091a47c694ead5ee5d08f7c5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f77fd2891e354952adb1369c7a84cbc3] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=36.8 K 2023-04-12 13:54:57,740 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:54:57,748 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5755d14f782e4be18e9fd9782144f85f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681307695438 2023-04-12 13:54:57,749 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 9b6970db091a47c694ead5ee5d08f7c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681307695517 2023-04-12 13:54:57,749 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f77fd2891e354952adb1369c7a84cbc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307696672 2023-04-12 13:54:57,750 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:54:57,750 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:54:57,750 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,750 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4d2dcd2dbefa484b91ad518a4c05f4da, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/21dec43afcca4ee6a1bebcdb89b50113, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9770298171fa411290ef4ce9676e913e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/92325d2bcc9d46afb6e394dd4f76d535] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=48.6 K 2023-04-12 13:54:57,751 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4d2dcd2dbefa484b91ad518a4c05f4da, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681307694800 2023-04-12 13:54:57,751 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 21dec43afcca4ee6a1bebcdb89b50113, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681307695438 2023-04-12 13:54:57,752 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9770298171fa411290ef4ce9676e913e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681307695517 2023-04-12 13:54:57,752 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 92325d2bcc9d46afb6e394dd4f76d535, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307696672 2023-04-12 13:54:57,765 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:57,768 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#285 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:57,791 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/96371f55faa44f4abe55657725cc2c19 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/96371f55faa44f4abe55657725cc2c19 2023-04-12 13:54:57,796 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 96371f55faa44f4abe55657725cc2c19(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:57,796 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:57,796 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=12, startTime=1681307697737; duration=0sec 2023-04-12 13:54:57,796 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:54:57,796 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:54:57,797 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:54:57,797 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:54:57,797 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,797 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/27669ab654be4b4f8518a0b056450a58, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/44995ba99af749d4afc4f1880df3db7c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1e89ac65fdea4fbf83a5b4277e723563] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=36.8 K 2023-04-12 13:54:57,798 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 27669ab654be4b4f8518a0b056450a58, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681307695438 2023-04-12 13:54:57,800 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 44995ba99af749d4afc4f1880df3db7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681307695517 2023-04-12 13:54:57,800 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1e89ac65fdea4fbf83a5b4277e723563, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307696672 2023-04-12 13:54:57,803 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:57,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:57,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:57,807 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#286 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:54:57,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:57,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:57,807 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:57,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:57,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:57,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:57,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:57,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:57,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:57,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:57,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:57,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:57,816 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:57,816 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 13:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:57,816 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:57,817 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:57,817 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:57,817 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:54:57,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:57,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:57,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307757957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:57,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:57,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307757961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:58,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307758062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:58,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307758064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,191 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/4ed31351a4e84cb3928e0dbf6bda27d5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/4ed31351a4e84cb3928e0dbf6bda27d5 2023-04-12 13:54:58,196 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 4ed31351a4e84cb3928e0dbf6bda27d5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:58,196 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:58,196 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=13, startTime=1681307697737; duration=0sec 2023-04-12 13:54:58,196 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:58,218 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/14019e0f8220471e84d183c70c8a5848 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/14019e0f8220471e84d183c70c8a5848 2023-04-12 13:54:58,222 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 14019e0f8220471e84d183c70c8a5848(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:54:58,222 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:58,222 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=13, startTime=1681307697737; duration=0sec 2023-04-12 13:54:58,223 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:54:58,230 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/885af40985a14ff981c4c0e01d5cf942 2023-04-12 13:54:58,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:58,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307758265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307758267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:58,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307758567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:58,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307758570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,640 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/408cc3255855489b80f6648390fd9106 2023-04-12 13:54:58,653 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/bbbd81c339dc4de99b29bde3a11bb185 2023-04-12 13:54:58,661 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/885af40985a14ff981c4c0e01d5cf942 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/885af40985a14ff981c4c0e01d5cf942 2023-04-12 13:54:58,665 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/885af40985a14ff981c4c0e01d5cf942, entries=150, sequenceid=539, filesize=11.9 K 2023-04-12 13:54:58,666 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/408cc3255855489b80f6648390fd9106 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/408cc3255855489b80f6648390fd9106 2023-04-12 13:54:58,670 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/408cc3255855489b80f6648390fd9106, entries=150, sequenceid=539, filesize=11.9 K 2023-04-12 13:54:58,671 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/bbbd81c339dc4de99b29bde3a11bb185 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bbbd81c339dc4de99b29bde3a11bb185 2023-04-12 13:54:58,674 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bbbd81c339dc4de99b29bde3a11bb185, entries=150, sequenceid=539, filesize=11.9 K 2023-04-12 13:54:58,675 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 8e79b7aef32bdb05b28a7dce92e13c3a in 859ms, sequenceid=539, compaction requested=false 2023-04-12 13:54:58,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:54:58,675 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:58,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:54:58,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:54:58,675 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:54:58,676 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:54:58,677 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,677 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:54:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:58,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70520322[Count = 0] remaining members to acquire global barrier 2023-04-12 13:54:58,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:54:58,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,681 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:54:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:54:58,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:54:58,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:58,683 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:58,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:54:58,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:54:58,686 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:54:58,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:54:58,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:54:58,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:54:58,687 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:54:58,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:54:58,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:54:58,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:58,689 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:54:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:58,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:58,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:54:58,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:58,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:54:58,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:54:58,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:54:58,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:54:58,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:58,694 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:54:58,694 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:54:58,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:54:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:54:58,794 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:54:58,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:54:58,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:54:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:54:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:54:58,798 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:54:58,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:58,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:54:58,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:54:58,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:58,799 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,799 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:54:58,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:54:58,799 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:58,799 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:58,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:54:58,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:54:58,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:54:58,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:54:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:54:58,802 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. started... 2023-04-12 13:54:58,802 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:54:58,803 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:54:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:54:59,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. as already flushing 2023-04-12 13:54:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307699092, max timestamp=1681307699107Num uniques 859; 2023-04-12 13:54:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307699092, max timestamp=1681307699107Num uniques 859; 2023-04-12 13:54:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307699092, max timestamp=1681307699107Num uniques 859; 2023-04-12 13:54:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:54:59,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307759152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307759152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,214 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/c3707ee96dc8441e941c7b0285c22f48 2023-04-12 13:54:59,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307759257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307759257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,363 DEBUG [Thread-1433] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:58527 2023-04-12 13:54:59,363 DEBUG [Thread-1433] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:59,363 DEBUG [Thread-1437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:58527 2023-04-12 13:54:59,364 DEBUG [Thread-1437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:59,364 DEBUG [Thread-1439] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:58527 2023-04-12 13:54:59,364 DEBUG [Thread-1439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:59,366 DEBUG [Thread-1435] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:58527 2023-04-12 13:54:59,366 DEBUG [Thread-1435] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:59,366 DEBUG [Thread-1431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:58527 2023-04-12 13:54:59,366 DEBUG [Thread-1431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:54:59,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,461 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307759461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307759461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,625 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/6d927282e58d4080ae631073b686be73 2023-04-12 13:54:59,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:54:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41512 deadline: 1681307759764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:54:59,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41536 deadline: 1681307759764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,033 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/e3fc61fb0fea4ec38e52ed27d80cea59 2023-04-12 13:55:00,037 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/c3707ee96dc8441e941c7b0285c22f48 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c3707ee96dc8441e941c7b0285c22f48 2023-04-12 13:55:00,040 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c3707ee96dc8441e941c7b0285c22f48, entries=150, sequenceid=551, filesize=11.9 K 2023-04-12 13:55:00,041 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/6d927282e58d4080ae631073b686be73 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6d927282e58d4080ae631073b686be73 2023-04-12 13:55:00,045 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6d927282e58d4080ae631073b686be73, entries=150, sequenceid=551, filesize=11.9 K 2023-04-12 13:55:00,045 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/e3fc61fb0fea4ec38e52ed27d80cea59 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/e3fc61fb0fea4ec38e52ed27d80cea59 2023-04-12 13:55:00,048 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/e3fc61fb0fea4ec38e52ed27d80cea59, entries=150, sequenceid=551, filesize=11.9 K 2023-04-12 13:55:00,049 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 8e79b7aef32bdb05b28a7dce92e13c3a in 1247ms, sequenceid=551, compaction requested=true 2023-04-12 13:55:00,049 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:55:00,049 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:00,049 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:00,049 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:00,049 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,049 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:00,049 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:00,051 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,051 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:00,051 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,051 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c142cc1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:00,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:00,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:00,054 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:00,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:00,056 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:00,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:00,058 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:00,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:00,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:00,058 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:00,058 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:00,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:00,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:00,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:00,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:00,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:00,062 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:00,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:00,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:00,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:00,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:00,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:00,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:00,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:00,063 DEBUG [Thread-1430] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:00,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:00,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:00,063 DEBUG [Thread-1430] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:00,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:00,163 DEBUG [Thread-1430] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:00,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:55:00,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-12 13:55:00,268 DEBUG [Thread-1428] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:58527 2023-04-12 13:55:00,268 DEBUG [Thread-1428] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:00,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:55:00,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:00,269 DEBUG [Thread-1420] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:58527 2023-04-12 13:55:00,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:55:00,269 DEBUG [Thread-1420] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:00,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:00,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:55:00,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:00,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/9957e75f81e147898eede440495c9538 2023-04-12 13:55:01,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/46eb6adbd006496d8451fa93015ab85f 2023-04-12 13:55:01,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/bc70d496eb504ae7952f48e9f26c6b17 2023-04-12 13:55:01,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/9957e75f81e147898eede440495c9538 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9957e75f81e147898eede440495c9538 2023-04-12 13:55:01,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9957e75f81e147898eede440495c9538, entries=150, sequenceid=590, filesize=11.9 K 2023-04-12 13:55:01,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/46eb6adbd006496d8451fa93015ab85f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/46eb6adbd006496d8451fa93015ab85f 2023-04-12 13:55:01,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/46eb6adbd006496d8451fa93015ab85f, entries=150, sequenceid=590, filesize=11.9 K 2023-04-12 13:55:01,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/bc70d496eb504ae7952f48e9f26c6b17 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bc70d496eb504ae7952f48e9f26c6b17 2023-04-12 13:55:01,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bc70d496eb504ae7952f48e9f26c6b17, entries=150, sequenceid=590, filesize=11.9 K 2023-04-12 13:55:01,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~440.72 KB/451296, currentSize=0 B/0 for 8e79b7aef32bdb05b28a7dce92e13c3a in 1244ms, sequenceid=590, compaction requested=true 2023-04-12 13:55:01,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:55:01,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:01,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:01,512 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:01,512 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:01,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:01,513 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:01,513 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/A is initiating minor compaction (all files) 2023-04-12 13:55:01,513 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:01,513 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/A in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:01,513 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/B is initiating minor compaction (all files) 2023-04-12 13:55:01,514 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/4ed31351a4e84cb3928e0dbf6bda27d5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/885af40985a14ff981c4c0e01d5cf942, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c3707ee96dc8441e941c7b0285c22f48, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9957e75f81e147898eede440495c9538] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=48.8 K 2023-04-12 13:55:01,514 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/B in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:01,514 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/14019e0f8220471e84d183c70c8a5848, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/408cc3255855489b80f6648390fd9106, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6d927282e58d4080ae631073b686be73, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/46eb6adbd006496d8451fa93015ab85f] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=48.8 K 2023-04-12 13:55:01,514 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4ed31351a4e84cb3928e0dbf6bda27d5, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307696672 2023-04-12 13:55:01,514 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 14019e0f8220471e84d183c70c8a5848, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307696672 2023-04-12 13:55:01,514 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 885af40985a14ff981c4c0e01d5cf942, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681307697331 2023-04-12 13:55:01,514 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 408cc3255855489b80f6648390fd9106, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681307697331 2023-04-12 13:55:01,515 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c3707ee96dc8441e941c7b0285c22f48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681307697949 2023-04-12 13:55:01,515 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6d927282e58d4080ae631073b686be73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681307697949 2023-04-12 13:55:01,515 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 9957e75f81e147898eede440495c9538, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681307699150 2023-04-12 13:55:01,515 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 46eb6adbd006496d8451fa93015ab85f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681307699150 2023-04-12 13:55:01,522 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#B#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:01,523 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#A#compaction#296 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:01,930 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/2e39606fae0b473d88b00d7ae8bbbdc1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2e39606fae0b473d88b00d7ae8bbbdc1 2023-04-12 13:55:01,931 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/58a70c2a45e04c8c9abff45c5814f617 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/58a70c2a45e04c8c9abff45c5814f617 2023-04-12 13:55:01,938 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/A of 8e79b7aef32bdb05b28a7dce92e13c3a into 58a70c2a45e04c8c9abff45c5814f617(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:01,938 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/B of 8e79b7aef32bdb05b28a7dce92e13c3a into 2e39606fae0b473d88b00d7ae8bbbdc1(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:01,938 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:55:01,938 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:55:01,938 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/A, priority=12, startTime=1681307701512; duration=0sec 2023-04-12 13:55:01,938 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/B, priority=12, startTime=1681307701512; duration=0sec 2023-04-12 13:55:01,938 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:01,938 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:01,938 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:01,939 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:01,939 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 8e79b7aef32bdb05b28a7dce92e13c3a/C is initiating minor compaction (all files) 2023-04-12 13:55:01,939 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8e79b7aef32bdb05b28a7dce92e13c3a/C in TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:01,939 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/96371f55faa44f4abe55657725cc2c19, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bbbd81c339dc4de99b29bde3a11bb185, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/e3fc61fb0fea4ec38e52ed27d80cea59, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bc70d496eb504ae7952f48e9f26c6b17] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp, totalSize=48.8 K 2023-04-12 13:55:01,940 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 96371f55faa44f4abe55657725cc2c19, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1681307696672 2023-04-12 13:55:01,940 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bbbd81c339dc4de99b29bde3a11bb185, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681307697331 2023-04-12 13:55:01,940 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e3fc61fb0fea4ec38e52ed27d80cea59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681307697949 2023-04-12 13:55:01,941 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bc70d496eb504ae7952f48e9f26c6b17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681307699150 2023-04-12 13:55:01,948 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8e79b7aef32bdb05b28a7dce92e13c3a#C#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:02,355 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/82dcbef323fc4705b3906e3b70e0ab64 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/82dcbef323fc4705b3906e3b70e0ab64 2023-04-12 13:55:02,359 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8e79b7aef32bdb05b28a7dce92e13c3a/C of 8e79b7aef32bdb05b28a7dce92e13c3a into 82dcbef323fc4705b3906e3b70e0ab64(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:02,359 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:55:02,359 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a., storeName=8e79b7aef32bdb05b28a7dce92e13c3a/C, priority=12, startTime=1681307701512; duration=0sec 2023-04-12 13:55:02,359 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:07,586 DEBUG [Thread-1426] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:58527 2023-04-12 13:55:07,587 DEBUG [Thread-1426] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:07,633 DEBUG [Thread-1422] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:58527 2023-04-12 13:55:07,633 DEBUG [Thread-1422] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:07,653 DEBUG [Thread-1424] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:58527 2023-04-12 13:55:07,653 DEBUG [Thread-1424] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:07,653 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5371 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5542 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5556 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5437 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 5535 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 13:55:07,654 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:55:07,654 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:58527 2023-04-12 13:55:07,654 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:07,655 INFO [Listener at localhost/46495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 13:55:07,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 13:55:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:07,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 13:55:07,658 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307707658"}]},"ts":"1681307707658"} 2023-04-12 13:55:07,659 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 13:55:07,660 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 13:55:07,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, UNASSIGN}] 2023-04-12 13:55:07,662 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, UNASSIGN 2023-04-12 13:55:07,663 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=8e79b7aef32bdb05b28a7dce92e13c3a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:07,663 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307707663"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307707663"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307707663"}]},"ts":"1681307707663"} 2023-04-12 13:55:07,664 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:55:07,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 13:55:07,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:55:07,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8e79b7aef32bdb05b28a7dce92e13c3a, disabling compactions & flushes 2023-04-12 13:55:07,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:07,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. after waiting 0 ms 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:07,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8e79b7aef32bdb05b28a7dce92e13c3a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=A 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=B 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8e79b7aef32bdb05b28a7dce92e13c3a, store=C 2023-04-12 13:55:07,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:07,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 13:55:08,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/c8462bc96e0a420f9c2e154cfdb360cc 2023-04-12 13:55:08,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 13:55:08,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/50d858fa62934f33a9b363dc5e36497d 2023-04-12 13:55:08,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 13:55:09,040 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/166b72d2f1f14afba374d6674bcf6337 2023-04-12 13:55:09,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/A/c8462bc96e0a420f9c2e154cfdb360cc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8462bc96e0a420f9c2e154cfdb360cc 2023-04-12 13:55:09,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8462bc96e0a420f9c2e154cfdb360cc, entries=100, sequenceid=599, filesize=9.5 K 2023-04-12 13:55:09,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/B/50d858fa62934f33a9b363dc5e36497d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/50d858fa62934f33a9b363dc5e36497d 2023-04-12 13:55:09,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/50d858fa62934f33a9b363dc5e36497d, entries=100, sequenceid=599, filesize=9.5 K 2023-04-12 13:55:09,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/.tmp/C/166b72d2f1f14afba374d6674bcf6337 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/166b72d2f1f14afba374d6674bcf6337 2023-04-12 13:55:09,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/166b72d2f1f14afba374d6674bcf6337, entries=100, sequenceid=599, filesize=9.5 K 2023-04-12 13:55:09,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 8e79b7aef32bdb05b28a7dce92e13c3a in 1240ms, sequenceid=599, compaction requested=false 2023-04-12 13:55:09,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a47ea6879284a9180281b69ba870147, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a85179c8598419d8da9c404a0a9e0aa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/eacb97d9a7e84e62b2da7759708f25c0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3bc52315936c4ef29634df7ceec2e491, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/01a4bd723a104aa8be35370b4e6ef779, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e05982eb98724704a2a12566f73ffd40, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f09eb6323839446e83dcfdba6f6888d5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a378b443f0da4a1cbfcea8c74d6fee24, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0a37c33b7e844b528301a54bb00ddaa8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3f59febea83f4c438e45f1659f57ac87, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/d778939a11d04a00b7fa20ff906cf878, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a9314260f8974f388fdcfea4f9ebea44, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/be33e7443012488295618bfa2f2d85b3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/93f2463c6e4945cb8aaf3309cccbb34a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/792e05da71d54335af40d27618704934, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/cecd7bfecdae49b89485d4ad1f8e4329, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/833910cbc7234c059233810ff437cd42, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e0ca51d9d1eb48e8a7f9c4234e139420, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3309ebb0f34b47ad8b4636d87f894a5c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8749f586bc34ae4bd8cf13c3e32ca46, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46dfacdf536c4f1f985744bd78c66f8c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6f70178ee9ca47249116d108f41aeecd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2a03555c8b724529a3af2b9a34d69836, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6a0997a071d64228b42fffc2dbed2eba, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5bb71a6d5f3c47a991efa1ec82fdca26, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46ae3b163fbe4c0ebbfec7af3ad5374e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0ad540149030456bb99c2f50b1d0e753, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e5a0797800e645c388cac10434c89331, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/de27cb59108b4a54ae030b2ff8facc96, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/8017a9c48a324c8694993e1abf881aa2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/91700dba6f1648c0a9d68c71253a791c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2648689433634c32bf19abc17611f5c7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5755d14f782e4be18e9fd9782144f85f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9b6970db091a47c694ead5ee5d08f7c5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/4ed31351a4e84cb3928e0dbf6bda27d5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f77fd2891e354952adb1369c7a84cbc3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/885af40985a14ff981c4c0e01d5cf942, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c3707ee96dc8441e941c7b0285c22f48, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9957e75f81e147898eede440495c9538] to archive 2023-04-12 13:55:09,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:55:09,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a47ea6879284a9180281b69ba870147 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a47ea6879284a9180281b69ba870147 2023-04-12 13:55:09,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a85179c8598419d8da9c404a0a9e0aa to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5a85179c8598419d8da9c404a0a9e0aa 2023-04-12 13:55:09,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/eacb97d9a7e84e62b2da7759708f25c0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/eacb97d9a7e84e62b2da7759708f25c0 2023-04-12 13:55:09,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3bc52315936c4ef29634df7ceec2e491 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3bc52315936c4ef29634df7ceec2e491 2023-04-12 13:55:09,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/01a4bd723a104aa8be35370b4e6ef779 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/01a4bd723a104aa8be35370b4e6ef779 2023-04-12 13:55:09,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e05982eb98724704a2a12566f73ffd40 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e05982eb98724704a2a12566f73ffd40 2023-04-12 13:55:09,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f09eb6323839446e83dcfdba6f6888d5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f09eb6323839446e83dcfdba6f6888d5 2023-04-12 13:55:09,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a378b443f0da4a1cbfcea8c74d6fee24 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a378b443f0da4a1cbfcea8c74d6fee24 2023-04-12 13:55:09,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/bf43b7fd4f5e4e3b9d07a0f6ecb8f63f 2023-04-12 13:55:09,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0a37c33b7e844b528301a54bb00ddaa8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0a37c33b7e844b528301a54bb00ddaa8 2023-04-12 13:55:09,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3f59febea83f4c438e45f1659f57ac87 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3f59febea83f4c438e45f1659f57ac87 2023-04-12 13:55:09,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/d778939a11d04a00b7fa20ff906cf878 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/d778939a11d04a00b7fa20ff906cf878 2023-04-12 13:55:09,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a9314260f8974f388fdcfea4f9ebea44 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/a9314260f8974f388fdcfea4f9ebea44 2023-04-12 13:55:09,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/be33e7443012488295618bfa2f2d85b3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/be33e7443012488295618bfa2f2d85b3 2023-04-12 13:55:09,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/93f2463c6e4945cb8aaf3309cccbb34a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/93f2463c6e4945cb8aaf3309cccbb34a 2023-04-12 13:55:09,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/792e05da71d54335af40d27618704934 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/792e05da71d54335af40d27618704934 2023-04-12 13:55:09,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/cecd7bfecdae49b89485d4ad1f8e4329 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/cecd7bfecdae49b89485d4ad1f8e4329 2023-04-12 13:55:09,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/833910cbc7234c059233810ff437cd42 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/833910cbc7234c059233810ff437cd42 2023-04-12 13:55:09,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e0ca51d9d1eb48e8a7f9c4234e139420 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e0ca51d9d1eb48e8a7f9c4234e139420 2023-04-12 13:55:09,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3309ebb0f34b47ad8b4636d87f894a5c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/3309ebb0f34b47ad8b4636d87f894a5c 2023-04-12 13:55:09,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8749f586bc34ae4bd8cf13c3e32ca46 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8749f586bc34ae4bd8cf13c3e32ca46 2023-04-12 13:55:09,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46dfacdf536c4f1f985744bd78c66f8c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46dfacdf536c4f1f985744bd78c66f8c 2023-04-12 13:55:09,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6f70178ee9ca47249116d108f41aeecd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6f70178ee9ca47249116d108f41aeecd 2023-04-12 13:55:09,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2a03555c8b724529a3af2b9a34d69836 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2a03555c8b724529a3af2b9a34d69836 2023-04-12 13:55:09,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6a0997a071d64228b42fffc2dbed2eba to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/6a0997a071d64228b42fffc2dbed2eba 2023-04-12 13:55:09,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5bb71a6d5f3c47a991efa1ec82fdca26 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5bb71a6d5f3c47a991efa1ec82fdca26 2023-04-12 13:55:09,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46ae3b163fbe4c0ebbfec7af3ad5374e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/46ae3b163fbe4c0ebbfec7af3ad5374e 2023-04-12 13:55:09,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0ad540149030456bb99c2f50b1d0e753 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/0ad540149030456bb99c2f50b1d0e753 2023-04-12 13:55:09,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e5a0797800e645c388cac10434c89331 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/e5a0797800e645c388cac10434c89331 2023-04-12 13:55:09,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/de27cb59108b4a54ae030b2ff8facc96 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/de27cb59108b4a54ae030b2ff8facc96 2023-04-12 13:55:09,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/8017a9c48a324c8694993e1abf881aa2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/8017a9c48a324c8694993e1abf881aa2 2023-04-12 13:55:09,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/91700dba6f1648c0a9d68c71253a791c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/91700dba6f1648c0a9d68c71253a791c 2023-04-12 13:55:09,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2648689433634c32bf19abc17611f5c7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/2648689433634c32bf19abc17611f5c7 2023-04-12 13:55:09,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5755d14f782e4be18e9fd9782144f85f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/5755d14f782e4be18e9fd9782144f85f 2023-04-12 13:55:09,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9b6970db091a47c694ead5ee5d08f7c5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9b6970db091a47c694ead5ee5d08f7c5 2023-04-12 13:55:09,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/4ed31351a4e84cb3928e0dbf6bda27d5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/4ed31351a4e84cb3928e0dbf6bda27d5 2023-04-12 13:55:09,110 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f77fd2891e354952adb1369c7a84cbc3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/f77fd2891e354952adb1369c7a84cbc3 2023-04-12 13:55:09,111 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/885af40985a14ff981c4c0e01d5cf942 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/885af40985a14ff981c4c0e01d5cf942 2023-04-12 13:55:09,112 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c3707ee96dc8441e941c7b0285c22f48 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c3707ee96dc8441e941c7b0285c22f48 2023-04-12 13:55:09,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9957e75f81e147898eede440495c9538 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/9957e75f81e147898eede440495c9538 2023-04-12 13:55:09,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/fa67792fef1e432ead4960ab03e31653, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/afc0c108c5f64a54b414656b4f5d241b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/cc621af8e2aa420f9b157bbbee47184d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba993c300de14cf285f8f16a93af4b94, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ca210053df8d4998b9abda8c55b3a48e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9a4396684d5f4515bff593628b21bd26, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bec12d3c8b5e4037ba7f3b20ad48617b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/86cbb0a491dc4fdfa7be191c76ee2c8e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3e5b4c6f894143f8bc890986ee03f5bb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2cf856ae33894f3192221126fcf7fa74, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b1dac72523f14d86a5a2c288f0cec1dd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e80ec15417894c22a56a5e1bdbd65506, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/8b7bb347844641f69f1efe0e2a2ad52c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e118cb1b3bb84ddf90f61bee46a984df, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba2110052c4d4f5d813363d162fb67d3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3b87fa0b4e464f6f8f5290f0a4353bf5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b13f1c4d0503461395ac26dc3a593dcf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/5b0853f0cf024b9c95199f73e3b3bff9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2eac5ee3a8404fe08ba1576819f1cedd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/639f2f7d57b74d9b9630621c072d4f4b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bb05fe081ef14913bf8a4cb8509fce45, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e911344d3e864dcb94a26488066e16c2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9af4675aadd14ea9943e0b6f9c31e45d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1255ace0a8c341b79d745c8324d8d6cd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9c13964f71374f7d897aa5b3cddc570b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/04ee87dd51014bcf8676eb0275203349, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/929ad4e91bbc4ea7b603cc432911f0d4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/be429db5a4e6436684de5683685147ca, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/005eab84b6c5443bb030609a8514e487, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6e032311857f4f60a59c25616896c038, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/d3dd9dd4b279407fb01af858b66a69a5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/27669ab654be4b4f8518a0b056450a58, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bc4278f989b24538aec54d24f92253ea, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/44995ba99af749d4afc4f1880df3db7c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/14019e0f8220471e84d183c70c8a5848, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1e89ac65fdea4fbf83a5b4277e723563, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/408cc3255855489b80f6648390fd9106, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6d927282e58d4080ae631073b686be73, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/46eb6adbd006496d8451fa93015ab85f] to archive 2023-04-12 13:55:09,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:55:09,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/fa67792fef1e432ead4960ab03e31653 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/fa67792fef1e432ead4960ab03e31653 2023-04-12 13:55:09,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/afc0c108c5f64a54b414656b4f5d241b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/afc0c108c5f64a54b414656b4f5d241b 2023-04-12 13:55:09,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/cc621af8e2aa420f9b157bbbee47184d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/cc621af8e2aa420f9b157bbbee47184d 2023-04-12 13:55:09,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba993c300de14cf285f8f16a93af4b94 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba993c300de14cf285f8f16a93af4b94 2023-04-12 13:55:09,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ca210053df8d4998b9abda8c55b3a48e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ca210053df8d4998b9abda8c55b3a48e 2023-04-12 13:55:09,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9a4396684d5f4515bff593628b21bd26 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9a4396684d5f4515bff593628b21bd26 2023-04-12 13:55:09,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/c5c5953ec1124ff5b8e4ffb1d0f2f56a 2023-04-12 13:55:09,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bec12d3c8b5e4037ba7f3b20ad48617b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bec12d3c8b5e4037ba7f3b20ad48617b 2023-04-12 13:55:09,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/86cbb0a491dc4fdfa7be191c76ee2c8e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/86cbb0a491dc4fdfa7be191c76ee2c8e 2023-04-12 13:55:09,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3e5b4c6f894143f8bc890986ee03f5bb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3e5b4c6f894143f8bc890986ee03f5bb 2023-04-12 13:55:09,127 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2cf856ae33894f3192221126fcf7fa74 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2cf856ae33894f3192221126fcf7fa74 2023-04-12 13:55:09,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b1dac72523f14d86a5a2c288f0cec1dd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b1dac72523f14d86a5a2c288f0cec1dd 2023-04-12 13:55:09,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e80ec15417894c22a56a5e1bdbd65506 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e80ec15417894c22a56a5e1bdbd65506 2023-04-12 13:55:09,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/8b7bb347844641f69f1efe0e2a2ad52c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/8b7bb347844641f69f1efe0e2a2ad52c 2023-04-12 13:55:09,130 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e118cb1b3bb84ddf90f61bee46a984df to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e118cb1b3bb84ddf90f61bee46a984df 2023-04-12 13:55:09,131 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba2110052c4d4f5d813363d162fb67d3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/ba2110052c4d4f5d813363d162fb67d3 2023-04-12 13:55:09,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3b87fa0b4e464f6f8f5290f0a4353bf5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/3b87fa0b4e464f6f8f5290f0a4353bf5 2023-04-12 13:55:09,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b13f1c4d0503461395ac26dc3a593dcf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/b13f1c4d0503461395ac26dc3a593dcf 2023-04-12 13:55:09,134 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/5b0853f0cf024b9c95199f73e3b3bff9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/5b0853f0cf024b9c95199f73e3b3bff9 2023-04-12 13:55:09,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2eac5ee3a8404fe08ba1576819f1cedd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2eac5ee3a8404fe08ba1576819f1cedd 2023-04-12 13:55:09,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/639f2f7d57b74d9b9630621c072d4f4b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/639f2f7d57b74d9b9630621c072d4f4b 2023-04-12 13:55:09,136 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bb05fe081ef14913bf8a4cb8509fce45 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bb05fe081ef14913bf8a4cb8509fce45 2023-04-12 13:55:09,137 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e911344d3e864dcb94a26488066e16c2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/e911344d3e864dcb94a26488066e16c2 2023-04-12 13:55:09,138 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9af4675aadd14ea9943e0b6f9c31e45d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9af4675aadd14ea9943e0b6f9c31e45d 2023-04-12 13:55:09,139 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1255ace0a8c341b79d745c8324d8d6cd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1255ace0a8c341b79d745c8324d8d6cd 2023-04-12 13:55:09,140 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9c13964f71374f7d897aa5b3cddc570b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/9c13964f71374f7d897aa5b3cddc570b 2023-04-12 13:55:09,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/04ee87dd51014bcf8676eb0275203349 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/04ee87dd51014bcf8676eb0275203349 2023-04-12 13:55:09,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/929ad4e91bbc4ea7b603cc432911f0d4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/929ad4e91bbc4ea7b603cc432911f0d4 2023-04-12 13:55:09,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/be429db5a4e6436684de5683685147ca to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/be429db5a4e6436684de5683685147ca 2023-04-12 13:55:09,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/005eab84b6c5443bb030609a8514e487 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/005eab84b6c5443bb030609a8514e487 2023-04-12 13:55:09,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6e032311857f4f60a59c25616896c038 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6e032311857f4f60a59c25616896c038 2023-04-12 13:55:09,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/d3dd9dd4b279407fb01af858b66a69a5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/d3dd9dd4b279407fb01af858b66a69a5 2023-04-12 13:55:09,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/27669ab654be4b4f8518a0b056450a58 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/27669ab654be4b4f8518a0b056450a58 2023-04-12 13:55:09,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bc4278f989b24538aec54d24f92253ea to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/bc4278f989b24538aec54d24f92253ea 2023-04-12 13:55:09,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/44995ba99af749d4afc4f1880df3db7c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/44995ba99af749d4afc4f1880df3db7c 2023-04-12 13:55:09,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/14019e0f8220471e84d183c70c8a5848 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/14019e0f8220471e84d183c70c8a5848 2023-04-12 13:55:09,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1e89ac65fdea4fbf83a5b4277e723563 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/1e89ac65fdea4fbf83a5b4277e723563 2023-04-12 13:55:09,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/408cc3255855489b80f6648390fd9106 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/408cc3255855489b80f6648390fd9106 2023-04-12 13:55:09,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6d927282e58d4080ae631073b686be73 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/6d927282e58d4080ae631073b686be73 2023-04-12 13:55:09,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/46eb6adbd006496d8451fa93015ab85f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/46eb6adbd006496d8451fa93015ab85f 2023-04-12 13:55:09,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/c57600d9ef8d42c1a32eaf1ed8bca5e4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b82af56af245491eb05943713c60ae65, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b5007f079f5f4f0eb68073a50ebf9414, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/37fffa21ff0f43ffbf84d55e9eac0577, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/28f09e16b95045b69e4ea4ef48210d01, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/0d69fdc9459244d0986a39ec87d5b38e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/f206eb15db8a4dc6a84045181973cfd7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/378d61b1f76b4999b739523fdeaf4d50, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b44354102ca841dd90e503517c862ed1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b96e960dfe704b0bbaeaaf52d28ddd4e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2411325824a349a8925b74db1dd28acf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/883cab695e9642449e11fc04a7a3a8e9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d6b52a739d4644a79cb0731a72ea8aa3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/80bc785d74974730a5108671da4a06f3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d9e174319da84c43be5a34326183509c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ad22268d7c6942baa9a1c4be1110695b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4789bd92be614be18cd3602d7a7059e1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/67ece128c57f4db9b79ee37a1528fe7a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/284cd10347bc4d6db462509aad85ce66, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/add08a5a0a3a414a8cf70e5abfca1f2d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b26805cbf22f4603b0ad64790f7ba3ce, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4ee6b08318da4dbdaf1cb3f4456713fd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d96a076c57244266b4960ee13aa633f8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/55b256e8146f42f5804a5263a2d6fdb6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/7cc4ebd9ac334bc4abb2832210422669, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/adff3bded3da4c7d8ec328d36ca5943f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2f93a2dded3247bba9c37c04435d3a52, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ea7b5220637947349bc49ed789b8f307, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9656b85379124e3dafa17373cd34ec01, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/5a29d83023f74bcc823b24e22bd7d115, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4d2dcd2dbefa484b91ad518a4c05f4da, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/27ac97b1692a4c6ea4944f6cfaadd037, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/21dec43afcca4ee6a1bebcdb89b50113, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9770298171fa411290ef4ce9676e913e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/96371f55faa44f4abe55657725cc2c19, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/92325d2bcc9d46afb6e394dd4f76d535, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bbbd81c339dc4de99b29bde3a11bb185, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/e3fc61fb0fea4ec38e52ed27d80cea59, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bc70d496eb504ae7952f48e9f26c6b17] to archive 2023-04-12 13:55:09,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:55:09,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/c57600d9ef8d42c1a32eaf1ed8bca5e4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/c57600d9ef8d42c1a32eaf1ed8bca5e4 2023-04-12 13:55:09,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b82af56af245491eb05943713c60ae65 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b82af56af245491eb05943713c60ae65 2023-04-12 13:55:09,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b5007f079f5f4f0eb68073a50ebf9414 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b5007f079f5f4f0eb68073a50ebf9414 2023-04-12 13:55:09,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/37fffa21ff0f43ffbf84d55e9eac0577 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/37fffa21ff0f43ffbf84d55e9eac0577 2023-04-12 13:55:09,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/28f09e16b95045b69e4ea4ef48210d01 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/28f09e16b95045b69e4ea4ef48210d01 2023-04-12 13:55:09,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/0d69fdc9459244d0986a39ec87d5b38e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/0d69fdc9459244d0986a39ec87d5b38e 2023-04-12 13:55:09,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/f206eb15db8a4dc6a84045181973cfd7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/f206eb15db8a4dc6a84045181973cfd7 2023-04-12 13:55:09,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/378d61b1f76b4999b739523fdeaf4d50 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/378d61b1f76b4999b739523fdeaf4d50 2023-04-12 13:55:09,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b44354102ca841dd90e503517c862ed1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b44354102ca841dd90e503517c862ed1 2023-04-12 13:55:09,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b96e960dfe704b0bbaeaaf52d28ddd4e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b96e960dfe704b0bbaeaaf52d28ddd4e 2023-04-12 13:55:09,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2411325824a349a8925b74db1dd28acf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2411325824a349a8925b74db1dd28acf 2023-04-12 13:55:09,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/883cab695e9642449e11fc04a7a3a8e9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/883cab695e9642449e11fc04a7a3a8e9 2023-04-12 13:55:09,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d6b52a739d4644a79cb0731a72ea8aa3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d6b52a739d4644a79cb0731a72ea8aa3 2023-04-12 13:55:09,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/80bc785d74974730a5108671da4a06f3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/80bc785d74974730a5108671da4a06f3 2023-04-12 13:55:09,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d9e174319da84c43be5a34326183509c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d9e174319da84c43be5a34326183509c 2023-04-12 13:55:09,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ad22268d7c6942baa9a1c4be1110695b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ad22268d7c6942baa9a1c4be1110695b 2023-04-12 13:55:09,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4789bd92be614be18cd3602d7a7059e1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4789bd92be614be18cd3602d7a7059e1 2023-04-12 13:55:09,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/67ece128c57f4db9b79ee37a1528fe7a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/67ece128c57f4db9b79ee37a1528fe7a 2023-04-12 13:55:09,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/284cd10347bc4d6db462509aad85ce66 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/284cd10347bc4d6db462509aad85ce66 2023-04-12 13:55:09,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/add08a5a0a3a414a8cf70e5abfca1f2d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/add08a5a0a3a414a8cf70e5abfca1f2d 2023-04-12 13:55:09,174 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b26805cbf22f4603b0ad64790f7ba3ce to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/b26805cbf22f4603b0ad64790f7ba3ce 2023-04-12 13:55:09,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4ee6b08318da4dbdaf1cb3f4456713fd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4ee6b08318da4dbdaf1cb3f4456713fd 2023-04-12 13:55:09,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d96a076c57244266b4960ee13aa633f8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/d96a076c57244266b4960ee13aa633f8 2023-04-12 13:55:09,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/55b256e8146f42f5804a5263a2d6fdb6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/55b256e8146f42f5804a5263a2d6fdb6 2023-04-12 13:55:09,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/7cc4ebd9ac334bc4abb2832210422669 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/7cc4ebd9ac334bc4abb2832210422669 2023-04-12 13:55:09,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/adff3bded3da4c7d8ec328d36ca5943f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/adff3bded3da4c7d8ec328d36ca5943f 2023-04-12 13:55:09,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2f93a2dded3247bba9c37c04435d3a52 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/2f93a2dded3247bba9c37c04435d3a52 2023-04-12 13:55:09,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ea7b5220637947349bc49ed789b8f307 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/ea7b5220637947349bc49ed789b8f307 2023-04-12 13:55:09,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9656b85379124e3dafa17373cd34ec01 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9656b85379124e3dafa17373cd34ec01 2023-04-12 13:55:09,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/5a29d83023f74bcc823b24e22bd7d115 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/5a29d83023f74bcc823b24e22bd7d115 2023-04-12 13:55:09,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4d2dcd2dbefa484b91ad518a4c05f4da to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/4d2dcd2dbefa484b91ad518a4c05f4da 2023-04-12 13:55:09,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/27ac97b1692a4c6ea4944f6cfaadd037 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/27ac97b1692a4c6ea4944f6cfaadd037 2023-04-12 13:55:09,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/21dec43afcca4ee6a1bebcdb89b50113 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/21dec43afcca4ee6a1bebcdb89b50113 2023-04-12 13:55:09,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9770298171fa411290ef4ce9676e913e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/9770298171fa411290ef4ce9676e913e 2023-04-12 13:55:09,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/96371f55faa44f4abe55657725cc2c19 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/96371f55faa44f4abe55657725cc2c19 2023-04-12 13:55:09,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/92325d2bcc9d46afb6e394dd4f76d535 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/92325d2bcc9d46afb6e394dd4f76d535 2023-04-12 13:55:09,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bbbd81c339dc4de99b29bde3a11bb185 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bbbd81c339dc4de99b29bde3a11bb185 2023-04-12 13:55:09,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/e3fc61fb0fea4ec38e52ed27d80cea59 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/e3fc61fb0fea4ec38e52ed27d80cea59 2023-04-12 13:55:09,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bc70d496eb504ae7952f48e9f26c6b17 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/bc70d496eb504ae7952f48e9f26c6b17 2023-04-12 13:55:09,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/recovered.edits/602.seqid, newMaxSeqId=602, maxSeqId=1 2023-04-12 13:55:09,195 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a. 2023-04-12 13:55:09,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8e79b7aef32bdb05b28a7dce92e13c3a: 2023-04-12 13:55:09,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:55:09,197 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=8e79b7aef32bdb05b28a7dce92e13c3a, regionState=CLOSED 2023-04-12 13:55:09,197 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307709197"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307709197"}]},"ts":"1681307709197"} 2023-04-12 13:55:09,198 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-12 13:55:09,199 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 8e79b7aef32bdb05b28a7dce92e13c3a, server=jenkins-hbase4.apache.org,35663,1681307616979 in 1.5340 sec 2023-04-12 13:55:09,200 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-12 13:55:09,200 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8e79b7aef32bdb05b28a7dce92e13c3a, UNASSIGN in 1.5380 sec 2023-04-12 13:55:09,200 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307709200"}]},"ts":"1681307709200"} 2023-04-12 13:55:09,201 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 13:55:09,202 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 13:55:09,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5480 sec 2023-04-12 13:55:09,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-12 13:55:09,762 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-12 13:55:09,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 13:55:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,765 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-12 13:55:09,765 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,769 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:55:09,771 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/recovered.edits] 2023-04-12 13:55:09,773 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/58a70c2a45e04c8c9abff45c5814f617 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/58a70c2a45e04c8c9abff45c5814f617 2023-04-12 13:55:09,774 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8462bc96e0a420f9c2e154cfdb360cc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/A/c8462bc96e0a420f9c2e154cfdb360cc 2023-04-12 13:55:09,776 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2e39606fae0b473d88b00d7ae8bbbdc1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/2e39606fae0b473d88b00d7ae8bbbdc1 2023-04-12 13:55:09,777 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/50d858fa62934f33a9b363dc5e36497d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/B/50d858fa62934f33a9b363dc5e36497d 2023-04-12 13:55:09,779 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/166b72d2f1f14afba374d6674bcf6337 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/166b72d2f1f14afba374d6674bcf6337 2023-04-12 13:55:09,780 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/82dcbef323fc4705b3906e3b70e0ab64 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/C/82dcbef323fc4705b3906e3b70e0ab64 2023-04-12 13:55:09,782 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/recovered.edits/602.seqid to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a/recovered.edits/602.seqid 2023-04-12 13:55:09,782 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/8e79b7aef32bdb05b28a7dce92e13c3a 2023-04-12 13:55:09,782 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:55:09,784 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,785 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 13:55:09,787 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 13:55:09,788 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,788 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 13:55:09,788 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681307709788"}]},"ts":"9223372036854775807"} 2023-04-12 13:55:09,789 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 13:55:09,789 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8e79b7aef32bdb05b28a7dce92e13c3a, NAME => 'TestAcidGuarantees,,1681307678195.8e79b7aef32bdb05b28a7dce92e13c3a.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 13:55:09,789 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 13:55:09,789 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681307709789"}]},"ts":"9223372036854775807"} 2023-04-12 13:55:09,790 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 13:55:09,792 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,793 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-04-12 13:55:09,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-12 13:55:09,867 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-12 13:55:09,878 INFO [Listener at localhost/46495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 233), OpenFileDescriptor=531 (was 547), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=366 (was 484), ProcessCount=171 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=288 (was 388) 2023-04-12 13:55:09,889 INFO [Listener at localhost/46495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=366, ProcessCount=171, AvailableMemoryMB=288 2023-04-12 13:55:09,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:55:09,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:55:09,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:09,894 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:55:09,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-12 13:55:09,894 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:55:09,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 13:55:09,895 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:09,896 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 empty. 2023-04-12 13:55:09,896 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:09,896 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:55:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 13:55:10,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 13:55:10,304 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:55:10,306 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7eb722b369ab23b3fdff931ce8c3cac2, NAME => 'TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:55:10,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 13:55:10,712 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:55:10,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 7eb722b369ab23b3fdff931ce8c3cac2, disabling compactions & flushes 2023-04-12 13:55:10,713 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:10,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:10,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. after waiting 0 ms 2023-04-12 13:55:10,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:10,713 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:10,713 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:10,715 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:55:10,716 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307710716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307710716"}]},"ts":"1681307710716"} 2023-04-12 13:55:10,717 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:55:10,717 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:55:10,718 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307710717"}]},"ts":"1681307710717"} 2023-04-12 13:55:10,718 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 13:55:10,722 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, ASSIGN}] 2023-04-12 13:55:10,723 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, ASSIGN 2023-04-12 13:55:10,724 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:55:10,875 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:10,875 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307710875"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307710875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307710875"}]},"ts":"1681307710875"} 2023-04-12 13:55:10,877 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:55:10,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 13:55:11,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:11,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7eb722b369ab23b3fdff931ce8c3cac2, NAME => 'TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:55:11,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:55:11,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,033 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,034 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A 2023-04-12 13:55:11,034 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A 2023-04-12 13:55:11,034 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:11,035 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7eb722b369ab23b3fdff931ce8c3cac2 columnFamilyName A 2023-04-12 13:55:11,035 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(310): Store=7eb722b369ab23b3fdff931ce8c3cac2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:11,035 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,036 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B 2023-04-12 13:55:11,036 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B 2023-04-12 13:55:11,036 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:11,037 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7eb722b369ab23b3fdff931ce8c3cac2 columnFamilyName B 2023-04-12 13:55:11,037 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(310): Store=7eb722b369ab23b3fdff931ce8c3cac2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:11,037 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,038 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C 2023-04-12 13:55:11,038 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C 2023-04-12 13:55:11,038 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:11,038 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7eb722b369ab23b3fdff931ce8c3cac2 columnFamilyName C 2023-04-12 13:55:11,039 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(310): Store=7eb722b369ab23b3fdff931ce8c3cac2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:11,039 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:11,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:55:11,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:11,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:55:11,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7eb722b369ab23b3fdff931ce8c3cac2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65724156, jitterRate=-0.0206337571144104}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:55:11,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:11,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., pid=37, masterSystemTime=1681307711028 2023-04-12 13:55:11,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:11,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:11,046 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:11,046 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307711046"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307711046"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307711046"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307711046"}]},"ts":"1681307711046"} 2023-04-12 13:55:11,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-12 13:55:11,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 in 171 msec 2023-04-12 13:55:11,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-12 13:55:11,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, ASSIGN in 326 msec 2023-04-12 13:55:11,050 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:55:11,050 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307711050"}]},"ts":"1681307711050"} 2023-04-12 13:55:11,051 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 13:55:11,053 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:55:11,053 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-12 13:55:11,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 13:55:11,999 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-12 13:55:12,000 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,005 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,006 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,008 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,009 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:55:12,010 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:55:12,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:55:12,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:55:12,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:12,422 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 13:55:12,423 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:55:12,423 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 13:55:12,425 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-12 13:55:12,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, REOPEN/MOVE}] 2023-04-12 13:55:12,427 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, REOPEN/MOVE 2023-04-12 13:55:12,428 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:12,428 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307712428"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307712428"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307712428"}]},"ts":"1681307712428"} 2023-04-12 13:55:12,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:55:12,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7eb722b369ab23b3fdff931ce8c3cac2, disabling compactions & flushes 2023-04-12 13:55:12,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. after waiting 0 ms 2023-04-12 13:55:12,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,584 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 13:55:12,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:12,585 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 7eb722b369ab23b3fdff931ce8c3cac2 to self. 2023-04-12 13:55:12,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,587 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=CLOSED 2023-04-12 13:55:12,587 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307712587"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307712587"}]},"ts":"1681307712587"} 2023-04-12 13:55:12,589 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-12 13:55:12,589 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 in 159 msec 2023-04-12 13:55:12,589 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=true 2023-04-12 13:55:12,740 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:12,740 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307712740"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307712740"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307712740"}]},"ts":"1681307712740"} 2023-04-12 13:55:12,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:55:12,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,895 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7eb722b369ab23b3fdff931ce8c3cac2, NAME => 'TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:55:12,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:55:12,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,897 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,898 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A 2023-04-12 13:55:12,898 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A 2023-04-12 13:55:12,898 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:12,898 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7eb722b369ab23b3fdff931ce8c3cac2 columnFamilyName A 2023-04-12 13:55:12,899 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(310): Store=7eb722b369ab23b3fdff931ce8c3cac2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:12,900 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,900 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B 2023-04-12 13:55:12,900 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B 2023-04-12 13:55:12,900 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:12,901 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7eb722b369ab23b3fdff931ce8c3cac2 columnFamilyName B 2023-04-12 13:55:12,901 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(310): Store=7eb722b369ab23b3fdff931ce8c3cac2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:12,901 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,902 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C 2023-04-12 13:55:12,902 DEBUG [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C 2023-04-12 13:55:12,902 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:12,902 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7eb722b369ab23b3fdff931ce8c3cac2 columnFamilyName C 2023-04-12 13:55:12,902 INFO [StoreOpener-7eb722b369ab23b3fdff931ce8c3cac2-1] regionserver.HStore(310): Store=7eb722b369ab23b3fdff931ce8c3cac2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:12,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:55:12,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:12,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7eb722b369ab23b3fdff931ce8c3cac2; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66116277, jitterRate=-0.014790698885917664}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:55:12,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:12,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., pid=42, masterSystemTime=1681307712892 2023-04-12 13:55:12,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,909 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:12,910 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307712909"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307712909"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307712909"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307712909"}]},"ts":"1681307712909"} 2023-04-12 13:55:12,911 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-12 13:55:12,911 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 in 169 msec 2023-04-12 13:55:12,912 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-12 13:55:12,913 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, REOPEN/MOVE in 484 msec 2023-04-12 13:55:12,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-12 13:55:12,914 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-12 13:55:12,915 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-12 13:55:12,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-12 13:55:12,917 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,921 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,921 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,926 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,926 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,929 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,930 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,933 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,933 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,936 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,937 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,941 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,941 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,945 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,945 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,951 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,951 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,954 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,954 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:12,958 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:12,964 DEBUG [hconnection-0x519b6d31-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:12,965 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,969 DEBUG [hconnection-0x6577236d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:12,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:12,971 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:12,971 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:12,971 DEBUG [hconnection-0x153d899e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,971 DEBUG [hconnection-0x454fe31b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,971 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:12,973 DEBUG [hconnection-0x2532aa09-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,972 DEBUG [hconnection-0x5b6956c2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,972 DEBUG [hconnection-0x1ad5bd98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,972 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60790, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,972 DEBUG [hconnection-0x4d22a7c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,975 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,976 DEBUG [hconnection-0x1b9c6994-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,973 DEBUG [hconnection-0x25fd5dc5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:12,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:12,976 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60826, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:12,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:12,976 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,978 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60854, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,976 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,980 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:12,980 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:12,981 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60820, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:12,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:12,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:12,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:12,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:12,989 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60838, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:12,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:12,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:12,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:12,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:12,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:12,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:12,999 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:13,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:13,005 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:55:13,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:13,027 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412520e76708d56457680d28152c0768dca to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412520e76708d56457680d28152c0768dca 2023-04-12 13:55:13,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307773025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307773025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307773026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307773027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307773028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307773129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307773129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307773129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307773129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307773132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307773332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307773332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307773333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307773333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307773334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,437 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/50a2545fda2f4d6cbe530a075a73a2de 2023-04-12 13:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307773637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307773637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307773637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307773638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:13,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307773638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:13,848 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/63e369b188da4c538468c589f952ba27 2023-04-12 13:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:14,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307774138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:14,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307774139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:14,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307774139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:14,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307774143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:14,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307774145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,259 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/fa62044639bb408093c8ac6b90014436 2023-04-12 13:55:14,264 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/50a2545fda2f4d6cbe530a075a73a2de as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/50a2545fda2f4d6cbe530a075a73a2de 2023-04-12 13:55:14,267 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/50a2545fda2f4d6cbe530a075a73a2de, entries=200, sequenceid=17, filesize=31.9 K 2023-04-12 13:55:14,268 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/63e369b188da4c538468c589f952ba27 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/63e369b188da4c538468c589f952ba27 2023-04-12 13:55:14,271 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/63e369b188da4c538468c589f952ba27, entries=150, sequenceid=17, filesize=11.6 K 2023-04-12 13:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/fa62044639bb408093c8ac6b90014436 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/fa62044639bb408093c8ac6b90014436 2023-04-12 13:55:14,276 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/fa62044639bb408093c8ac6b90014436, entries=150, sequenceid=17, filesize=11.6 K 2023-04-12 13:55:14,276 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1272ms, sequenceid=17, compaction requested=false 2023-04-12 13:55:14,276 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 13:55:14,277 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:14,277 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:14,277 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:14,279 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:14,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:14,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:14,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c5febe[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:14,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:14,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,282 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:14,283 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:14,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:14,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:14,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:14,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:14,286 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:14,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:14,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:14,286 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:14,286 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:14,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:14,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:14,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:14,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,293 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,293 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:14,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:14,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:14,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:14,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:14,293 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:14,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:14,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:14,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:14,293 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:14,294 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:14,294 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:14,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:14,394 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:14,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:14,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:14,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:14,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:14,400 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:14,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:14,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:14,400 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:14,401 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,401 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:14,402 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:14,402 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:14,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:14,403 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:14,403 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:14,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:14,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:14,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:14,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:14,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:14,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:14,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:14,404 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:14,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:14,728 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 13:55:14,820 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041265aa166d2a7d44b791a85a38102f1d15 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041265aa166d2a7d44b791a85a38102f1d15 2023-04-12 13:55:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:15,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:15,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307775155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307775155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307775155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307775157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307775158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,226 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/4a0cc2b8844241a69a3e583ce3517c65 2023-04-12 13:55:15,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307775260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307775260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307775262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307775264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307775467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307775467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307775468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307775468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,638 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/50721cbe55ca44d0bf92ee40f5dc6400 2023-04-12 13:55:15,652 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/78e98b57ba964f58b562b330eb10ed15 2023-04-12 13:55:15,656 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/4a0cc2b8844241a69a3e583ce3517c65 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4a0cc2b8844241a69a3e583ce3517c65 2023-04-12 13:55:15,659 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4a0cc2b8844241a69a3e583ce3517c65, entries=150, sequenceid=36, filesize=25.1 K 2023-04-12 13:55:15,660 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/50721cbe55ca44d0bf92ee40f5dc6400 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/50721cbe55ca44d0bf92ee40f5dc6400 2023-04-12 13:55:15,663 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/50721cbe55ca44d0bf92ee40f5dc6400, entries=150, sequenceid=36, filesize=11.6 K 2023-04-12 13:55:15,671 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/78e98b57ba964f58b562b330eb10ed15 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/78e98b57ba964f58b562b330eb10ed15 2023-04-12 13:55:15,678 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/78e98b57ba964f58b562b330eb10ed15, entries=150, sequenceid=36, filesize=11.6 K 2023-04-12 13:55:15,679 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1275ms, sequenceid=36, compaction requested=false 2023-04-12 13:55:15,679 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:15,679 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:15,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:15,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:15,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:15,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:15,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,681 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:15,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7889a482[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:15,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:15,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:15,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:15,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:15,686 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,686 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:15,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:15,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:15,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:15,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:15,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:15,693 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:15,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:15,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:15,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:15,693 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:15,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,698 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:15,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:15,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:15,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:15,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:15,701 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:15,702 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:15,702 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:15,778 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:55:15,778 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:15,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:15,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:15,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:15,779 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:15,779 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:15,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:15,802 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:15,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:15,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:15,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:15,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:15,825 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:15,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:15,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:15,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:15,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,831 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:15,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:15,831 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,831 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,832 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:15,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:15,834 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b9c0d8c8b3584c0eaa25ea837b5ed0ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b9c0d8c8b3584c0eaa25ea837b5ed0ab 2023-04-12 13:55:15,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:15,836 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:15,836 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:15,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,836 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:15,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307775824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,836 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:15,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,838 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307775831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307775832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:15,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1822f3a2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:15,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:15,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,842 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:15,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:15,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:15,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:15,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:15,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:15,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307775837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:15,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:15,849 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:15,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:15,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:15,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:15,849 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:15,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:15,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,852 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:15,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:15,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:15,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:15,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:15,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:15,855 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:15,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:15,855 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:15,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:15,862 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/6b14d410d0ff46e4b4bbc1873776332f 2023-04-12 13:55:15,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307775938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307775940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307775941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:15,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307775949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,964 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:15,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:15,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:15,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:15,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:15,968 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:15,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:15,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:15,968 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:15,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,970 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:15,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:15,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:15,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:15,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:15,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:15,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:15,971 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:15,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:15,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:15,973 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:15,973 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:15,973 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:15,974 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:15,977 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,977 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:15,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652db643[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:15,979 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:15,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:15,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:15,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:15,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:15,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:15,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:15,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:15,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:15,983 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:15,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:15,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:15,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:15,983 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:15,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:15,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:15,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:15,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,986 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:15,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:15,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:15,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:15,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:15,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:15,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:15,992 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:15,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:15,992 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:15,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:15,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:15,993 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:15,993 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:16,093 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:16,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:16,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:16,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:16,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:16,097 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,097 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,099 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:16,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:16,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:16,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:16,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:16,102 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:16,102 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:16,103 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,103 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:16,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f279456[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:16,105 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:16,105 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,106 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,106 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,106 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:16,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:16,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:16,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:16,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:16,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:16,111 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:16,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:16,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:16,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:16,111 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:16,113 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,113 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:16,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,113 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,117 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:16,117 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:16,117 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,117 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:16,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:16,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307776146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307776146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307776147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307776153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,217 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:16,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:16,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:16,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:16,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:16,221 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,222 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,222 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,223 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,223 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:16,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:16,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:16,224 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,224 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:16,224 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:16,224 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:16,227 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,227 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,227 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,227 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:16,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6abf9684[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:16,229 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:16,229 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,233 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:16,233 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,233 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:16,234 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:16,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:16,236 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:16,236 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:16,236 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:16,237 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:16,237 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:16,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,240 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:16,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:16,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,244 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:16,244 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:16,244 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:16,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/85892f549be6484ebf92472a962ca452 2023-04-12 13:55:16,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/42e99fe4715647b99b83bf190eec9db8 2023-04-12 13:55:16,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/6b14d410d0ff46e4b4bbc1873776332f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6b14d410d0ff46e4b4bbc1873776332f 2023-04-12 13:55:16,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6b14d410d0ff46e4b4bbc1873776332f, entries=200, sequenceid=50, filesize=31.9 K 2023-04-12 13:55:16,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/85892f549be6484ebf92472a962ca452 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/85892f549be6484ebf92472a962ca452 2023-04-12 13:55:16,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/85892f549be6484ebf92472a962ca452, entries=150, sequenceid=50, filesize=11.6 K 2023-04-12 13:55:16,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/42e99fe4715647b99b83bf190eec9db8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/42e99fe4715647b99b83bf190eec9db8 2023-04-12 13:55:16,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/42e99fe4715647b99b83bf190eec9db8, entries=150, sequenceid=50, filesize=11.6 K 2023-04-12 13:55:16,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7eb722b369ab23b3fdff931ce8c3cac2 in 546ms, sequenceid=50, compaction requested=true 2023-04-12 13:55:16,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:16,324 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:16,324 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:16,324 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:16,324 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:16,324 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:16,327 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91118 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:16,327 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:16,328 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/A is initiating minor compaction (all files) 2023-04-12 13:55:16,328 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/B is initiating minor compaction (all files) 2023-04-12 13:55:16,328 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/A in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,328 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/B in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,328 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/50a2545fda2f4d6cbe530a075a73a2de, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4a0cc2b8844241a69a3e583ce3517c65, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6b14d410d0ff46e4b4bbc1873776332f] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=89.0 K 2023-04-12 13:55:16,328 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/63e369b188da4c538468c589f952ba27, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/50721cbe55ca44d0bf92ee40f5dc6400, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/85892f549be6484ebf92472a962ca452] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=34.8 K 2023-04-12 13:55:16,328 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 50a2545fda2f4d6cbe530a075a73a2de, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681307712999 2023-04-12 13:55:16,328 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 63e369b188da4c538468c589f952ba27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681307713001 2023-04-12 13:55:16,328 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4a0cc2b8844241a69a3e583ce3517c65, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681307713009 2023-04-12 13:55:16,329 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 50721cbe55ca44d0bf92ee40f5dc6400, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681307713009 2023-04-12 13:55:16,329 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6b14d410d0ff46e4b4bbc1873776332f, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681307715152 2023-04-12 13:55:16,330 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 85892f549be6484ebf92472a962ca452, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681307715152 2023-04-12 13:55:16,337 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#B#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:16,344 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#A#compaction#311 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:16,347 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:16,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:16,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:16,359 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/f66de41b98d64479b28829304d1f98f9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/f66de41b98d64479b28829304d1f98f9 2023-04-12 13:55:16,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:16,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:16,362 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:16,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:16,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:16,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,364 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,364 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,365 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,365 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:16,366 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/B of 7eb722b369ab23b3fdff931ce8c3cac2 into f66de41b98d64479b28829304d1f98f9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:16,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:16,367 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/B, priority=13, startTime=1681307716324; duration=0sec 2023-04-12 13:55:16,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:16,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:16,368 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:16,368 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/C is initiating minor compaction (all files) 2023-04-12 13:55:16,368 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/C in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,368 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/fa62044639bb408093c8ac6b90014436, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/78e98b57ba964f58b562b330eb10ed15, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/42e99fe4715647b99b83bf190eec9db8] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=34.8 K 2023-04-12 13:55:16,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:16,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,368 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting fa62044639bb408093c8ac6b90014436, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681307713001 2023-04-12 13:55:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:16,369 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 78e98b57ba964f58b562b330eb10ed15, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681307713009 2023-04-12 13:55:16,369 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 42e99fe4715647b99b83bf190eec9db8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681307715152 2023-04-12 13:55:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:16,371 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:16,371 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,371 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:16,371 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:55:16,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:16,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:16,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:16,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:16,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:16,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:16,381 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#C#compaction#313 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:16,415 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/19d1dd58a90f49c0b9b686425a33fbd0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/19d1dd58a90f49c0b9b686425a33fbd0 2023-04-12 13:55:16,420 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/C of 7eb722b369ab23b3fdff931ce8c3cac2 into 19d1dd58a90f49c0b9b686425a33fbd0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:16,420 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:16,420 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/C, priority=13, startTime=1681307716324; duration=0sec 2023-04-12 13:55:16,420 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:16,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:16,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:16,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307776468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307776469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307776470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307776474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307776575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307776575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307776576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307776577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307776780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307776781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307776782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:16,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307776783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,805 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/34e388254ed94cbb86c77e4dd688cd34 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/34e388254ed94cbb86c77e4dd688cd34 2023-04-12 13:55:16,809 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/A of 7eb722b369ab23b3fdff931ce8c3cac2 into 34e388254ed94cbb86c77e4dd688cd34(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:16,809 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:16,809 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/A, priority=13, startTime=1681307716324; duration=0sec 2023-04-12 13:55:16,809 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:16,817 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121ef1f879292b45a48e1b07fb1b1b3052 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121ef1f879292b45a48e1b07fb1b1b3052 2023-04-12 13:55:16,827 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/a228fde2e1f44d13a74ebe8cfbfed487 2023-04-12 13:55:16,841 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/21f0449168c04306ba7619f098e6b149 2023-04-12 13:55:16,850 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/10f9041a452e411c8c737ef63c1a001a 2023-04-12 13:55:16,854 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/a228fde2e1f44d13a74ebe8cfbfed487 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a228fde2e1f44d13a74ebe8cfbfed487 2023-04-12 13:55:16,857 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a228fde2e1f44d13a74ebe8cfbfed487, entries=150, sequenceid=69, filesize=25.1 K 2023-04-12 13:55:16,857 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/21f0449168c04306ba7619f098e6b149 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/21f0449168c04306ba7619f098e6b149 2023-04-12 13:55:16,861 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/21f0449168c04306ba7619f098e6b149, entries=150, sequenceid=69, filesize=11.6 K 2023-04-12 13:55:16,862 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/10f9041a452e411c8c737ef63c1a001a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/10f9041a452e411c8c737ef63c1a001a 2023-04-12 13:55:16,865 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/10f9041a452e411c8c737ef63c1a001a, entries=150, sequenceid=69, filesize=11.6 K 2023-04-12 13:55:16,865 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7eb722b369ab23b3fdff931ce8c3cac2 in 494ms, sequenceid=69, compaction requested=false 2023-04-12 13:55:16,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:16,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:16,870 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,870 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658fd792[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:16,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:16,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,873 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,873 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:16,873 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:16,873 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,873 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:16,873 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:16,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:16,875 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:16,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:16,878 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:16,878 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:16,878 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:16,880 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,880 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,880 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:16,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:16,881 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:16,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:16,884 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,884 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:16,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:16,884 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:16,884 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:16,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:16,985 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:16,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:16,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:16,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:16,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:16,988 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:16,988 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,988 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:16,988 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:16,988 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,988 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,989 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:16,990 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:16,990 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:16,990 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:16,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:16,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:16,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:16,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:16,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:16,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:16,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:16,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:16,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:16,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:16,992 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:16,992 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:17,015 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412deaae8414cbb4ae59ea44440d85b31de to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412deaae8414cbb4ae59ea44440d85b31de 2023-04-12 13:55:17,027 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/bf31869d71ed43ce9daeda7ff8dd339f 2023-04-12 13:55:17,047 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/8fe2e3f8baa34ab0be520b70707e2377 2023-04-12 13:55:17,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:17,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:17,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307777109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307777110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307777112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307777117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307777163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,168 DEBUG [Thread-2236] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:55:17,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307777219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307777219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307777219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307777219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307777424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307777424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307777424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307777425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,459 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/80f517b7997d4f7bb13ef6ba15c0e174 2023-04-12 13:55:17,463 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/bf31869d71ed43ce9daeda7ff8dd339f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/bf31869d71ed43ce9daeda7ff8dd339f 2023-04-12 13:55:17,467 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/bf31869d71ed43ce9daeda7ff8dd339f, entries=150, sequenceid=85, filesize=25.1 K 2023-04-12 13:55:17,468 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/8fe2e3f8baa34ab0be520b70707e2377 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8fe2e3f8baa34ab0be520b70707e2377 2023-04-12 13:55:17,471 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8fe2e3f8baa34ab0be520b70707e2377, entries=150, sequenceid=85, filesize=11.6 K 2023-04-12 13:55:17,472 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/80f517b7997d4f7bb13ef6ba15c0e174 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/80f517b7997d4f7bb13ef6ba15c0e174 2023-04-12 13:55:17,478 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/80f517b7997d4f7bb13ef6ba15c0e174, entries=150, sequenceid=85, filesize=11.6 K 2023-04-12 13:55:17,479 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7eb722b369ab23b3fdff931ce8c3cac2 in 487ms, sequenceid=85, compaction requested=true 2023-04-12 13:55:17,479 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:17,479 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:17,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:17,481 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,481 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:17,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d98eace[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:17,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:17,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,485 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:17,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:17,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:17,486 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:17,486 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:17,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:17,490 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:17,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:17,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:17,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:17,490 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:17,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:17,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:17,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:17,494 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,498 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:17,498 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:17,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:17,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:17,498 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:17,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:17,499 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:17,499 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:17,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:17,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:17,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:17,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:17,599 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:17,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:17,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:17,603 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:17,603 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:17,603 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:17,603 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:17,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:17,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,605 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:17,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:17,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:17,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:17,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:17,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:17,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:17,607 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:17,607 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:17,607 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:55:17,607 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:17,607 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:17,607 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:17,608 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:17,608 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:17,608 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:17,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:17,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307777741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307777746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307777746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307777747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307777848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307777853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307777853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:17,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:17,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307777853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,030 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412f64e66695af84b5192ce9bd5f456003d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f64e66695af84b5192ce9bd5f456003d 2023-04-12 13:55:18,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307778051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307778057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307778057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307778058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307778356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307778359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307778361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307778364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,446 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/dc1ac3967a084154876e0361a9c7de9f 2023-04-12 13:55:18,457 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/553d8d4987f14594b18e129a44b79989 2023-04-12 13:55:18,467 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/cad51693c04246a58db1bfab1936942c 2023-04-12 13:55:18,471 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/dc1ac3967a084154876e0361a9c7de9f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/dc1ac3967a084154876e0361a9c7de9f 2023-04-12 13:55:18,475 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/dc1ac3967a084154876e0361a9c7de9f, entries=150, sequenceid=103, filesize=25.1 K 2023-04-12 13:55:18,476 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/553d8d4987f14594b18e129a44b79989 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/553d8d4987f14594b18e129a44b79989 2023-04-12 13:55:18,480 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/553d8d4987f14594b18e129a44b79989, entries=150, sequenceid=103, filesize=11.6 K 2023-04-12 13:55:18,481 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/cad51693c04246a58db1bfab1936942c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cad51693c04246a58db1bfab1936942c 2023-04-12 13:55:18,484 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cad51693c04246a58db1bfab1936942c, entries=150, sequenceid=103, filesize=11.6 K 2023-04-12 13:55:18,485 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7eb722b369ab23b3fdff931ce8c3cac2 in 878ms, sequenceid=103, compaction requested=true 2023-04-12 13:55:18,485 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:18,485 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:18,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:18,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:18,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:18,487 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:18,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,488 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:18,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:18,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@efe560f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:18,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:18,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,490 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:18,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:18,496 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:18,498 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:18,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:18,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:18,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:18,499 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:18,500 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:18,501 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,501 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:18,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:18,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:18,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:18,502 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:18,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:18,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:18,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:18,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:18,506 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:18,506 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:18,505 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:18,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:18,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,606 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:18,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:18,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:18,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:18,609 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:18,609 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:18,609 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:18,609 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:18,609 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,609 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:18,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,611 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:18,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:18,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:18,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:18,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:18,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:18,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:18,612 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:18,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:18,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:18,612 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:55:18,613 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:18,613 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:18,613 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:18,613 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:18,613 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:18,613 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:18,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:18,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:18,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307778893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307778907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307778907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:18,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:18,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307778908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307779009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307779018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307779019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307779019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,029 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304123bcdb61a33674658b846770bb8e82fc7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123bcdb61a33674658b846770bb8e82fc7 2023-04-12 13:55:19,036 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/675308ab8cd14a45a35fae02497cec79 2023-04-12 13:55:19,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307779215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307779226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307779226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307779226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,447 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/4b959efb851244389062b01c19be63ee 2023-04-12 13:55:19,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307779523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307779534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307779536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:19,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307779536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,894 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/cf4ccc21cfab4e9c86ad34dbe33774b4 2023-04-12 13:55:19,898 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/675308ab8cd14a45a35fae02497cec79 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/675308ab8cd14a45a35fae02497cec79 2023-04-12 13:55:19,901 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/675308ab8cd14a45a35fae02497cec79, entries=150, sequenceid=117, filesize=25.1 K 2023-04-12 13:55:19,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/4b959efb851244389062b01c19be63ee as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b959efb851244389062b01c19be63ee 2023-04-12 13:55:19,906 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b959efb851244389062b01c19be63ee, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:55:19,906 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/cf4ccc21cfab4e9c86ad34dbe33774b4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cf4ccc21cfab4e9c86ad34dbe33774b4 2023-04-12 13:55:19,911 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cf4ccc21cfab4e9c86ad34dbe33774b4, entries=150, sequenceid=117, filesize=11.6 K 2023-04-12 13:55:19,912 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1299ms, sequenceid=117, compaction requested=true 2023-04-12 13:55:19,912 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:19,912 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:19,913 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:19,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:19,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e3eae2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:19,915 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:19,915 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,917 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:19,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:19,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:19,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:19,921 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:19,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:19,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:19,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:19,921 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:19,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:19,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:19,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:19,924 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:19,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:19,924 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:19,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,925 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:19,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:19,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:19,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:19,928 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:19,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:19,928 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:19,928 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:19,928 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,929 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:19,929 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,029 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:20,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:20,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:20,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:55:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:20,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:20,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:20,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:20,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:20,034 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:20,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:20,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:20,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:20,034 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:20,034 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:20,037 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,038 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,038 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:20,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:20,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:20,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:20,039 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:20,039 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,039 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:20,039 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:20,039 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:20,039 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,039 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:20,040 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:20,040 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:20,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:20,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:20,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:20,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb9d2b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:20,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:20,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,050 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:20,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:20,052 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d816d33870b54db39367c53bfbf8ddf7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d816d33870b54db39367c53bfbf8ddf7 2023-04-12 13:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,052 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:20,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:20,056 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:20,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:20,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:20,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:20,056 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:20,058 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/b41993703c814b17a8ab7a61a8b54cb6 2023-04-12 13:55:20,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307780056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,074 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,074 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:20,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:20,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,074 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,074 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,074 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:20,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,074 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,074 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:20,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:20,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307780064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307780066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307780072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/6ea24902a19f46f18671affc7696e140 2023-04-12 13:55:20,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307780168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,175 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:20,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:20,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:20,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:20,178 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:20,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:20,178 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:20,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307780176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307780177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307780177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,180 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:20,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:20,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:20,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:20,182 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:20,183 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:20,185 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:20,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292fc906[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:20,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:20,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,189 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:20,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:20,193 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:20,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:20,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:20,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:20,193 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:20,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,202 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:20,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,203 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:20,203 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,303 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:20,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:20,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:20,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:20,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:20,306 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:20,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:20,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:20,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:20,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:20,310 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,310 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:20,310 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:20,310 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:20,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:20,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:20,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559dac02[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:20,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:20,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,316 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,316 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:20,320 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:20,320 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,322 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,322 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:20,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:20,324 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:20,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:20,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:20,324 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:20,324 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:20,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,326 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:20,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,331 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:20,331 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:20,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307780374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307780381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307780381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307780382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,432 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:20,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:20,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:20,436 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:20,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:20,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:20,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:20,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:20,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:20,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:20,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:20,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:20,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,443 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522ef8a9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:20,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:20,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:20,445 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,445 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:20,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:20,447 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,447 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,447 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,447 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:20,449 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:20,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:20,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:20,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:20,449 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:20,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,452 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,452 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,453 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:20,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:20,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:20,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:20,456 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:20,456 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:20,456 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:20,456 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,456 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,456 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:20,456 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/64d432f8ef1f4df5837dc01b65b8fe30 2023-04-12 13:55:20,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/b41993703c814b17a8ab7a61a8b54cb6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/b41993703c814b17a8ab7a61a8b54cb6 2023-04-12 13:55:20,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/b41993703c814b17a8ab7a61a8b54cb6, entries=200, sequenceid=136, filesize=32.1 K 2023-04-12 13:55:20,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/6ea24902a19f46f18671affc7696e140 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6ea24902a19f46f18671affc7696e140 2023-04-12 13:55:20,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6ea24902a19f46f18671affc7696e140, entries=150, sequenceid=136, filesize=11.8 K 2023-04-12 13:55:20,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/64d432f8ef1f4df5837dc01b65b8fe30 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/64d432f8ef1f4df5837dc01b65b8fe30 2023-04-12 13:55:20,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/64d432f8ef1f4df5837dc01b65b8fe30, entries=150, sequenceid=136, filesize=11.8 K 2023-04-12 13:55:20,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 7eb722b369ab23b3fdff931ce8c3cac2 in 495ms, sequenceid=136, compaction requested=true 2023-04-12 13:55:20,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:20,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:20,529 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:55:20,529 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:55:20,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:20,529 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:20,531 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 161719 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:55:20,531 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/A is initiating minor compaction (all files) 2023-04-12 13:55:20,531 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:55:20,531 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/A in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,531 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/B is initiating minor compaction (all files) 2023-04-12 13:55:20,531 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/B in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,531 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/34e388254ed94cbb86c77e4dd688cd34, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a228fde2e1f44d13a74ebe8cfbfed487, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/bf31869d71ed43ce9daeda7ff8dd339f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/dc1ac3967a084154876e0361a9c7de9f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/675308ab8cd14a45a35fae02497cec79, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/b41993703c814b17a8ab7a61a8b54cb6] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=157.9 K 2023-04-12 13:55:20,531 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/f66de41b98d64479b28829304d1f98f9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/21f0449168c04306ba7619f098e6b149, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8fe2e3f8baa34ab0be520b70707e2377, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/553d8d4987f14594b18e129a44b79989, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b959efb851244389062b01c19be63ee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6ea24902a19f46f18671affc7696e140] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=69.9 K 2023-04-12 13:55:20,532 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 34e388254ed94cbb86c77e4dd688cd34, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681307715152 2023-04-12 13:55:20,532 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f66de41b98d64479b28829304d1f98f9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681307715152 2023-04-12 13:55:20,532 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a228fde2e1f44d13a74ebe8cfbfed487, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681307715808 2023-04-12 13:55:20,532 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 21f0449168c04306ba7619f098e6b149, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681307715808 2023-04-12 13:55:20,533 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bf31869d71ed43ce9daeda7ff8dd339f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681307716459 2023-04-12 13:55:20,533 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8fe2e3f8baa34ab0be520b70707e2377, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681307716459 2023-04-12 13:55:20,533 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 553d8d4987f14594b18e129a44b79989, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681307717108 2023-04-12 13:55:20,533 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting dc1ac3967a084154876e0361a9c7de9f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681307717108 2023-04-12 13:55:20,533 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4b959efb851244389062b01c19be63ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307717738 2023-04-12 13:55:20,535 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 675308ab8cd14a45a35fae02497cec79, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307717738 2023-04-12 13:55:20,535 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6ea24902a19f46f18671affc7696e140, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681307718890 2023-04-12 13:55:20,535 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b41993703c814b17a8ab7a61a8b54cb6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681307718890 2023-04-12 13:55:20,550 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#B#compaction#329 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:20,556 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:20,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:20,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:20,560 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:20,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:20,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:20,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:20,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:20,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:20,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:20,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:20,566 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#A#compaction#330 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:20,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:20,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:20,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:20,573 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:55:20,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:20,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:20,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:20,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:20,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:20,574 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:20,579 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/e71ed46191ac4169ab64a125fe328558 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e71ed46191ac4169ab64a125fe328558 2023-04-12 13:55:20,583 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/B of 7eb722b369ab23b3fdff931ce8c3cac2 into e71ed46191ac4169ab64a125fe328558(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:20,583 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:20,583 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/B, priority=10, startTime=1681307720529; duration=0sec 2023-04-12 13:55:20,583 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:20,583 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:55:20,586 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71569 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:55:20,586 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/C is initiating minor compaction (all files) 2023-04-12 13:55:20,586 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/C in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:20,586 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/19d1dd58a90f49c0b9b686425a33fbd0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/10f9041a452e411c8c737ef63c1a001a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/80f517b7997d4f7bb13ef6ba15c0e174, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cad51693c04246a58db1bfab1936942c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cf4ccc21cfab4e9c86ad34dbe33774b4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/64d432f8ef1f4df5837dc01b65b8fe30] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=69.9 K 2023-04-12 13:55:20,587 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 19d1dd58a90f49c0b9b686425a33fbd0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681307715152 2023-04-12 13:55:20,587 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 10f9041a452e411c8c737ef63c1a001a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681307715808 2023-04-12 13:55:20,587 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 80f517b7997d4f7bb13ef6ba15c0e174, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681307716459 2023-04-12 13:55:20,588 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting cad51693c04246a58db1bfab1936942c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681307717108 2023-04-12 13:55:20,588 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting cf4ccc21cfab4e9c86ad34dbe33774b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681307717738 2023-04-12 13:55:20,588 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 64d432f8ef1f4df5837dc01b65b8fe30, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681307718890 2023-04-12 13:55:20,619 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#C#compaction#332 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:20,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:20,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:20,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307780716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307780717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307780717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307780724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307780827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307780829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307780830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:20,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:20,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307780835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,012 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304125507d36172cb455dbe5ec5b7d57279b8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125507d36172cb455dbe5ec5b7d57279b8 2023-04-12 13:55:21,020 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/f416ff4fa4cd4158a9ef1b11df7e0496 2023-04-12 13:55:21,029 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/159afe661c254e5b982d9215e3412f96 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/159afe661c254e5b982d9215e3412f96 2023-04-12 13:55:21,035 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/A of 7eb722b369ab23b3fdff931ce8c3cac2 into 159afe661c254e5b982d9215e3412f96(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:21,035 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:21,035 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/A, priority=10, startTime=1681307720529; duration=0sec 2023-04-12 13:55:21,035 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/f3be966cf7d54208b6fda810a4cb94ee as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/f3be966cf7d54208b6fda810a4cb94ee 2023-04-12 13:55:21,035 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:21,039 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/C of 7eb722b369ab23b3fdff931ce8c3cac2 into f3be966cf7d54208b6fda810a4cb94ee(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:21,039 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:21,039 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/C, priority=10, startTime=1681307720529; duration=0sec 2023-04-12 13:55:21,039 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:21,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307781032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307781034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307781034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,041 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/74a854b1e752452fb9df613e8bfcd765 2023-04-12 13:55:21,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307781042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307781177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,181 DEBUG [Thread-2236] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:55:21,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307781341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307781342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307781343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307781348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,461 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/ba6bc4729a124ef5a34eeec3d00ec57f 2023-04-12 13:55:21,466 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/f416ff4fa4cd4158a9ef1b11df7e0496 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f416ff4fa4cd4158a9ef1b11df7e0496 2023-04-12 13:55:21,470 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f416ff4fa4cd4158a9ef1b11df7e0496, entries=150, sequenceid=149, filesize=25.3 K 2023-04-12 13:55:21,470 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/74a854b1e752452fb9df613e8bfcd765 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/74a854b1e752452fb9df613e8bfcd765 2023-04-12 13:55:21,475 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/74a854b1e752452fb9df613e8bfcd765, entries=150, sequenceid=149, filesize=11.8 K 2023-04-12 13:55:21,475 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/ba6bc4729a124ef5a34eeec3d00ec57f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ba6bc4729a124ef5a34eeec3d00ec57f 2023-04-12 13:55:21,479 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ba6bc4729a124ef5a34eeec3d00ec57f, entries=150, sequenceid=149, filesize=11.8 K 2023-04-12 13:55:21,482 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 7eb722b369ab23b3fdff931ce8c3cac2 in 909ms, sequenceid=149, compaction requested=false 2023-04-12 13:55:21,482 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:21,482 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:21,484 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d530502[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:21,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:21,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,489 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:21,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,491 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:21,491 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:21,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:21,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:21,496 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:21,496 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:21,496 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:21,496 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:21,496 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:21,498 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,498 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:21,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,498 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:21,511 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:21,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:21,511 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:21,511 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:21,511 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:21,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:21,612 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:21,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:21,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:21,615 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:21,615 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:21,615 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:21,615 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:21,616 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,616 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,617 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:21,617 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,617 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:21,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:21,619 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:21,633 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412096bf61e584a42a18e9cd4cb57ae83cc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412096bf61e584a42a18e9cd4cb57ae83cc 2023-04-12 13:55:21,640 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=171, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/910a3790d9074ee2931bc45fc4e14a0c 2023-04-12 13:55:21,652 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/93472f11491a48e7b22e5ae28b57fb04 2023-04-12 13:55:21,663 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/ddcb03097ca7435c84f508f22b09c1e1 2023-04-12 13:55:21,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/910a3790d9074ee2931bc45fc4e14a0c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/910a3790d9074ee2931bc45fc4e14a0c 2023-04-12 13:55:21,679 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/910a3790d9074ee2931bc45fc4e14a0c, entries=150, sequenceid=171, filesize=25.3 K 2023-04-12 13:55:21,680 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/93472f11491a48e7b22e5ae28b57fb04 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/93472f11491a48e7b22e5ae28b57fb04 2023-04-12 13:55:21,692 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/93472f11491a48e7b22e5ae28b57fb04, entries=150, sequenceid=171, filesize=11.8 K 2023-04-12 13:55:21,693 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/ddcb03097ca7435c84f508f22b09c1e1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ddcb03097ca7435c84f508f22b09c1e1 2023-04-12 13:55:21,705 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ddcb03097ca7435c84f508f22b09c1e1, entries=150, sequenceid=171, filesize=11.8 K 2023-04-12 13:55:21,706 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 7eb722b369ab23b3fdff931ce8c3cac2 in 87ms, sequenceid=171, compaction requested=true 2023-04-12 13:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:21,706 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:21,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:21,708 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,708 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,709 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,709 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:21,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a638a57[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:21,712 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:21,712 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:21,714 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:21,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:21,720 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:21,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:21,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:21,723 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:21,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:21,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:21,724 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:21,724 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:21,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:21,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,726 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,730 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,733 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:21,733 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:21,733 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:21,733 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:21,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:21,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:21,734 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:21,734 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:21,733 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,734 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:21,834 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:21,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:21,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:21,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:21,839 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:21,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:21,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:21,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:21,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:21,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:21,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:21,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:21,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,843 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:21,844 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:21,844 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:21,847 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:21,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a6d0a5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:21,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:21,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:21,852 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:21,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:21,855 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:21,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:21,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:21,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:21,855 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:21,857 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,857 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,857 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,858 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,859 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:21,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:21,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:21,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:21,863 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,863 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:21,864 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:21,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:21,935 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:55:21,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:21,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:21,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:21,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:21,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:21,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:21,966 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:21,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:21,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:21,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:21,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:21,982 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:21,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:21,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:21,982 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:21,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,983 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,984 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,984 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:21,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:21,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:21,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:21,986 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304128e36ec6738304472b82b1e9417388c6b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128e36ec6738304472b82b1e9417388c6b 2023-04-12 13:55:21,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:21,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:21,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:21,992 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:21,993 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,993 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:21,993 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:21,993 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:21,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,995 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e897e8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:21,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:21,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:21,999 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/525b11a9c4084d2f8bd64ebdd30c988c 2023-04-12 13:55:22,000 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:22,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:22,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:22,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:22,005 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:22,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:22,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:22,005 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:22,005 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:22,006 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,007 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,007 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,008 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307721989, max timestamp=1681307722011Num uniques 859; 2023-04-12 13:55:22,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:22,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:22,015 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:22,015 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307721989, max timestamp=1681307722011Num uniques 859; 2023-04-12 13:55:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307721989, max timestamp=1681307722011Num uniques 859; 2023-04-12 13:55:22,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:22,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/ad5ca44173bc4bb1ba2321547e3d84b0 2023-04-12 13:55:22,040 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/a57dbe39d2d745958f28aaaa1eb4ac7f 2023-04-12 13:55:22,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/525b11a9c4084d2f8bd64ebdd30c988c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/525b11a9c4084d2f8bd64ebdd30c988c 2023-04-12 13:55:22,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/525b11a9c4084d2f8bd64ebdd30c988c, entries=350, sequenceid=182, filesize=52.6 K 2023-04-12 13:55:22,050 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/ad5ca44173bc4bb1ba2321547e3d84b0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ad5ca44173bc4bb1ba2321547e3d84b0 2023-04-12 13:55:22,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ad5ca44173bc4bb1ba2321547e3d84b0, entries=150, sequenceid=182, filesize=11.8 K 2023-04-12 13:55:22,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/a57dbe39d2d745958f28aaaa1eb4ac7f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/a57dbe39d2d745958f28aaaa1eb4ac7f 2023-04-12 13:55:22,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/a57dbe39d2d745958f28aaaa1eb4ac7f, entries=150, sequenceid=182, filesize=11.8 K 2023-04-12 13:55:22,064 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 7eb722b369ab23b3fdff931ce8c3cac2 in 129ms, sequenceid=182, compaction requested=true 2023-04-12 13:55:22,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:22,064 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:22,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:22,065 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:22,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:22,065 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:22,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:22,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-12 13:55:22,067 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 131856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:22,067 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:22,067 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/A is initiating minor compaction (all files) 2023-04-12 13:55:22,067 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/B is initiating minor compaction (all files) 2023-04-12 13:55:22,067 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/A in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,068 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/B in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,068 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/159afe661c254e5b982d9215e3412f96, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f416ff4fa4cd4158a9ef1b11df7e0496, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/910a3790d9074ee2931bc45fc4e14a0c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/525b11a9c4084d2f8bd64ebdd30c988c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=128.8 K 2023-04-12 13:55:22,068 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e71ed46191ac4169ab64a125fe328558, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/74a854b1e752452fb9df613e8bfcd765, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/93472f11491a48e7b22e5ae28b57fb04, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ad5ca44173bc4bb1ba2321547e3d84b0] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=47.3 K 2023-04-12 13:55:22,068 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 159afe661c254e5b982d9215e3412f96, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681307718890 2023-04-12 13:55:22,068 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e71ed46191ac4169ab64a125fe328558, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681307718890 2023-04-12 13:55:22,069 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f416ff4fa4cd4158a9ef1b11df7e0496, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681307720045 2023-04-12 13:55:22,069 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 74a854b1e752452fb9df613e8bfcd765, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681307720045 2023-04-12 13:55:22,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681307722043, max timestamp=1681307722066Num uniques 1068; 2023-04-12 13:55:22,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 13:55:22,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:22,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:22,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:22,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:22,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:22,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:22,072 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 93472f11491a48e7b22e5ae28b57fb04, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681307720715 2023-04-12 13:55:22,072 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 910a3790d9074ee2931bc45fc4e14a0c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681307720715 2023-04-12 13:55:22,073 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 525b11a9c4084d2f8bd64ebdd30c988c, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307721893 2023-04-12 13:55:22,073 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ad5ca44173bc4bb1ba2321547e3d84b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307721925 2023-04-12 13:55:22,115 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:22,117 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#B#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:22,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:22,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:22,125 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:22,125 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,125 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:22,125 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:22,126 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,126 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,138 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,138 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,138 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,138 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:22,143 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121a2ef3e4501b4021833929e341a32239 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121a2ef3e4501b4021833929e341a32239 2023-04-12 13:55:22,143 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#A#compaction#343 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:22,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:22,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:22,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:22,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:22,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:22,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,150 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:22,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5286f102[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:22,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:22,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,153 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:22,154 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,154 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:22,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,156 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:22,162 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:22,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:22,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:22,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:22,162 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:22,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307782097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:22,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,173 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,173 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:22,173 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307782165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307782165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307782165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,273 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:22,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:22,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:22,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:22,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:22,277 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:22,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:22,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:22,278 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,278 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,282 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307782276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,283 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:22,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307782281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307782282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:22,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:22,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:22,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:22,291 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:22,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:22,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307782284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1471f837[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:22,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:22,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:22,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:22,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:22,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:22,303 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:22,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:22,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:22,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:22,303 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:22,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,310 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:22,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:22,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,311 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:22,311 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:22,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,311 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,411 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:22,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:22,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:22,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:22,414 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:22,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:22,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:22,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,416 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:22,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:22,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:22,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:22,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:22,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:22,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,420 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:22,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408631e4[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:22,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:22,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:22,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:22,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:22,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,426 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:22,428 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:22,428 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:22,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:22,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:22,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:22,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:22,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,437 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:22,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,437 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:22,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307782485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307782488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307782488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307782495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,537 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:22,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:22,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:22,541 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,541 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,544 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:22,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:22,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:22,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:22,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:22,546 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:22,546 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:22,546 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/6982987c663043a98bb35ab6dfd01a40 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6982987c663043a98bb35ab6dfd01a40 2023-04-12 13:55:22,550 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,550 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,550 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/B of 7eb722b369ab23b3fdff931ce8c3cac2 into 6982987c663043a98bb35ab6dfd01a40(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:22,550 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,551 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/B, priority=12, startTime=1681307722064; duration=0sec 2023-04-12 13:55:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:22,551 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:22,551 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,552 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:22,552 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/C is initiating minor compaction (all files) 2023-04-12 13:55:22,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:22,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d2b9cb[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:22,552 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/C in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:22,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,552 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/f3be966cf7d54208b6fda810a4cb94ee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ba6bc4729a124ef5a34eeec3d00ec57f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ddcb03097ca7435c84f508f22b09c1e1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/a57dbe39d2d745958f28aaaa1eb4ac7f] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=47.3 K 2023-04-12 13:55:22,553 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f3be966cf7d54208b6fda810a4cb94ee, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681307718890 2023-04-12 13:55:22,553 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ba6bc4729a124ef5a34eeec3d00ec57f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681307720045 2023-04-12 13:55:22,553 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,554 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ddcb03097ca7435c84f508f22b09c1e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681307720715 2023-04-12 13:55:22,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:22,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:22,554 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a57dbe39d2d745958f28aaaa1eb4ac7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307721925 2023-04-12 13:55:22,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:22,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:22,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:22,558 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:22,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:22,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:22,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:22,558 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:22,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:22,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,562 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/a7a88d8f996a4558abc75293cfc0a192 2023-04-12 13:55:22,562 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#C#compaction#344 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:22,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:22,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:22,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:22,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:22,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:22,574 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:22,574 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:22,580 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/bcebb911f18241459bcedc2b040abdf1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/bcebb911f18241459bcedc2b040abdf1 2023-04-12 13:55:22,584 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/C of 7eb722b369ab23b3fdff931ce8c3cac2 into bcebb911f18241459bcedc2b040abdf1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:22,584 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:22,584 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/C, priority=12, startTime=1681307722065; duration=0sec 2023-04-12 13:55:22,585 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:22,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/8639b86721fb46a0aa4a69e72bc221f8 2023-04-12 13:55:22,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/4c8e8112387d4b4d88a616181f9c9c52 2023-04-12 13:55:22,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/a7a88d8f996a4558abc75293cfc0a192 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a7a88d8f996a4558abc75293cfc0a192 2023-04-12 13:55:22,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a7a88d8f996a4558abc75293cfc0a192, entries=300, sequenceid=220, filesize=45.8 K 2023-04-12 13:55:22,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/8639b86721fb46a0aa4a69e72bc221f8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8639b86721fb46a0aa4a69e72bc221f8 2023-04-12 13:55:22,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8639b86721fb46a0aa4a69e72bc221f8, entries=150, sequenceid=220, filesize=11.8 K 2023-04-12 13:55:22,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/4c8e8112387d4b4d88a616181f9c9c52 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c8e8112387d4b4d88a616181f9c9c52 2023-04-12 13:55:22,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c8e8112387d4b4d88a616181f9c9c52, entries=150, sequenceid=220, filesize=11.8 K 2023-04-12 13:55:22,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~111.82 KB/114500, heapSize ~312.47 KB/319968, currentSize=73.80 KB/75570 for 7eb722b369ab23b3fdff931ce8c3cac2 in 560ms, sequenceid=220, compaction requested=false 2023-04-12 13:55:22,627 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:22,674 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:22,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:22,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:22,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:22,677 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:22,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:22,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:22,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:22,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:22,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:22,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:22,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:22,682 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:22,682 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:22,698 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412f534435702f84a3895b2f44787d65793 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f534435702f84a3895b2f44787d65793 2023-04-12 13:55:22,703 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=236, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/04c368c60c104c90a921201de48f8d8f 2023-04-12 13:55:22,711 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/39840b19e73f4ed4a66c8bd20792205b 2023-04-12 13:55:22,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:22,797 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:22,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307782822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307782823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307782824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307782833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60790 deadline: 1681307782934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60830 deadline: 1681307782934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307782934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:22,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60798 deadline: 1681307782943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:22,982 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/5fb2844bb2cb4b0a8bfa0b6268ec3819 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/5fb2844bb2cb4b0a8bfa0b6268ec3819 2023-04-12 13:55:22,986 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/A of 7eb722b369ab23b3fdff931ce8c3cac2 into 5fb2844bb2cb4b0a8bfa0b6268ec3819(size=25.7 K), total size for store is 71.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:22,986 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:22,987 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/A, priority=12, startTime=1681307722064; duration=0sec 2023-04-12 13:55:22,987 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:23,125 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/6b8dac50f2cf43af8ab4b7618368d9e3 2023-04-12 13:55:23,129 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/04c368c60c104c90a921201de48f8d8f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/04c368c60c104c90a921201de48f8d8f 2023-04-12 13:55:23,133 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/04c368c60c104c90a921201de48f8d8f, entries=150, sequenceid=236, filesize=25.3 K 2023-04-12 13:55:23,134 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/39840b19e73f4ed4a66c8bd20792205b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/39840b19e73f4ed4a66c8bd20792205b 2023-04-12 13:55:23,141 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/39840b19e73f4ed4a66c8bd20792205b, entries=150, sequenceid=236, filesize=11.8 K 2023-04-12 13:55:23,142 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/6b8dac50f2cf43af8ab4b7618368d9e3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6b8dac50f2cf43af8ab4b7618368d9e3 2023-04-12 13:55:23,150 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6b8dac50f2cf43af8ab4b7618368d9e3, entries=150, sequenceid=236, filesize=11.8 K 2023-04-12 13:55:23,151 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7eb722b369ab23b3fdff931ce8c3cac2 in 469ms, sequenceid=236, compaction requested=true 2023-04-12 13:55:23,151 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:23,151 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:23,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:23,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:23,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:23,154 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:23,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b95e52[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:23,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:23,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:23,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:23,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:23,166 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:23,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:23,171 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:23,171 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:23,171 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:23,171 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:23,171 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:23,172 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,172 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,172 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,173 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:23,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,178 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,178 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:23,178 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:23,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307783185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307783185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307783186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307783186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,278 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:23,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:23,282 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:23,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:23,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:23,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,284 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:23,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:23,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:23,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:23,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:23,286 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,286 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:23,286 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:23,286 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:23,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,288 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d12aaf2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:23,289 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:23,289 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:23,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:23,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:23,292 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:23,296 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:23,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:23,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:23,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:23,296 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60826 deadline: 1681307783293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307783293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:23,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:23,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,304 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,305 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:23,305 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:23,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307783299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307783299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,405 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:23,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:23,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:23,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:23,416 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:23,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:23,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:23,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,419 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:23,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:23,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:23,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:23,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:23,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6ab5a5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:23,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:23,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,429 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:23,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:23,435 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:23,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:23,438 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:23,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:23,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:23,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:23,438 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:23,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,446 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:23,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:23,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,446 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:23,446 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:23,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307783497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307783498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307783506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307783507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,546 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:23,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:23,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:23,549 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:23,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:23,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:23,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,551 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:23,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:23,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:23,560 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,560 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:23,560 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:23,560 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:23,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:23,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:23,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa04102[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:23,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,565 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,565 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:23,565 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:23,565 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:23,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:23,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:23,570 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:23,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:23,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:23,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:23,570 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:23,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,573 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:23,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:23,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:23,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:23,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:23,576 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:23,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:23,576 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:23,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:23,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,587 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d3ccae2a795141a3a781142d172d1e7c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d3ccae2a795141a3a781142d172d1e7c 2023-04-12 13:55:23,594 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/311e558ee7ee40c596023ef8aeb962be 2023-04-12 13:55:23,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/d3337e3844414cd68e745c38bbeee900 2023-04-12 13:55:23,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4 2023-04-12 13:55:23,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/311e558ee7ee40c596023ef8aeb962be as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/311e558ee7ee40c596023ef8aeb962be 2023-04-12 13:55:23,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/311e558ee7ee40c596023ef8aeb962be, entries=200, sequenceid=256, filesize=32.1 K 2023-04-12 13:55:23,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/d3337e3844414cd68e745c38bbeee900 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/d3337e3844414cd68e745c38bbeee900 2023-04-12 13:55:23,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/d3337e3844414cd68e745c38bbeee900, entries=150, sequenceid=256, filesize=11.8 K 2023-04-12 13:55:23,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4 2023-04-12 13:55:23,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4, entries=150, sequenceid=256, filesize=11.8 K 2023-04-12 13:55:23,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 7eb722b369ab23b3fdff931ce8c3cac2 in 477ms, sequenceid=256, compaction requested=true 2023-04-12 13:55:23,633 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:23,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:23,634 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:23,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:23,634 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:23,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:23,635 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 131992 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:23,636 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/A is initiating minor compaction (all files) 2023-04-12 13:55:23,636 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:23,636 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/A in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,636 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/B is initiating minor compaction (all files) 2023-04-12 13:55:23,636 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/B in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,636 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/5fb2844bb2cb4b0a8bfa0b6268ec3819, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a7a88d8f996a4558abc75293cfc0a192, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/04c368c60c104c90a921201de48f8d8f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/311e558ee7ee40c596023ef8aeb962be] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=128.9 K 2023-04-12 13:55:23,636 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6982987c663043a98bb35ab6dfd01a40, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8639b86721fb46a0aa4a69e72bc221f8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/39840b19e73f4ed4a66c8bd20792205b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/d3337e3844414cd68e745c38bbeee900] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=47.4 K 2023-04-12 13:55:23,636 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5fb2844bb2cb4b0a8bfa0b6268ec3819, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307721925 2023-04-12 13:55:23,636 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6982987c663043a98bb35ab6dfd01a40, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307721925 2023-04-12 13:55:23,637 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a7a88d8f996a4558abc75293cfc0a192, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681307722043 2023-04-12 13:55:23,637 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8639b86721fb46a0aa4a69e72bc221f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681307722045 2023-04-12 13:55:23,637 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 39840b19e73f4ed4a66c8bd20792205b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1681307722096 2023-04-12 13:55:23,637 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 04c368c60c104c90a921201de48f8d8f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1681307722096 2023-04-12 13:55:23,637 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d3337e3844414cd68e745c38bbeee900, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681307722808 2023-04-12 13:55:23,637 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 311e558ee7ee40c596023ef8aeb962be, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681307722808 2023-04-12 13:55:23,643 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#B#compaction#353 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:23,663 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/88df68f32b9a4153a17277367dee6d7e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/88df68f32b9a4153a17277367dee6d7e 2023-04-12 13:55:23,666 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#A#compaction#354 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:23,667 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/B of 7eb722b369ab23b3fdff931ce8c3cac2 into 88df68f32b9a4153a17277367dee6d7e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:23,667 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:23,667 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/B, priority=12, startTime=1681307723634; duration=0sec 2023-04-12 13:55:23,667 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:23,667 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:55:23,668 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:55:23,668 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/C is initiating minor compaction (all files) 2023-04-12 13:55:23,668 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/C in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,669 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/bcebb911f18241459bcedc2b040abdf1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c8e8112387d4b4d88a616181f9c9c52, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6b8dac50f2cf43af8ab4b7618368d9e3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=47.4 K 2023-04-12 13:55:23,669 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bcebb911f18241459bcedc2b040abdf1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1681307721925 2023-04-12 13:55:23,669 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4c8e8112387d4b4d88a616181f9c9c52, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681307722045 2023-04-12 13:55:23,669 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6b8dac50f2cf43af8ab4b7618368d9e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1681307722096 2023-04-12 13:55:23,670 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1a24df7d9e5b4ffbaeaf932b5f07ecb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681307722808 2023-04-12 13:55:23,677 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:23,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:23,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:23,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:23,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:23,682 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:23,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:23,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:23,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,683 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#C#compaction#355 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:23,683 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:23,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:23,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,684 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:23,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:23,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:23,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:23,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:23,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:23,685 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:23,685 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:23,686 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:55:23,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:23,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:23,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:23,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:23,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:23,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:23,695 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/14515f8dc8e24e1086c4a81ebc902058 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/14515f8dc8e24e1086c4a81ebc902058 2023-04-12 13:55:23,699 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/C of 7eb722b369ab23b3fdff931ce8c3cac2 into 14515f8dc8e24e1086c4a81ebc902058(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:23,699 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:23,699 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/C, priority=12, startTime=1681307723634; duration=0sec 2023-04-12 13:55:23,699 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:23,702 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304122e42a03892124c5dbc933f9e39a76beb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122e42a03892124c5dbc933f9e39a76beb 2023-04-12 13:55:23,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:23,801 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:23,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307783846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307783846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307783847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307783848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307783949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307783949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307783955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:23,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:23,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307783957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,116 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/6ff22e8312144150a5e5595a80eac4c6 2023-04-12 13:55:24,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307784155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307784156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307784163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307784165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307784460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307784462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307784470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307784473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,500 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/4f771522c9cd431e9133b56e1f017592 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4f771522c9cd431e9133b56e1f017592 2023-04-12 13:55:24,504 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/A of 7eb722b369ab23b3fdff931ce8c3cac2 into 4f771522c9cd431e9133b56e1f017592(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:24,504 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:24,504 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/A, priority=12, startTime=1681307723634; duration=0sec 2023-04-12 13:55:24,504 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:24,539 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/492a5a77fa3d400bbd80e56f5c7e2c90 2023-04-12 13:55:24,559 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/b4596b8fc36f43138b7522ceeeb69fb3 2023-04-12 13:55:24,563 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/6ff22e8312144150a5e5595a80eac4c6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6ff22e8312144150a5e5595a80eac4c6 2023-04-12 13:55:24,566 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6ff22e8312144150a5e5595a80eac4c6, entries=150, sequenceid=271, filesize=25.4 K 2023-04-12 13:55:24,567 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/492a5a77fa3d400bbd80e56f5c7e2c90 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/492a5a77fa3d400bbd80e56f5c7e2c90 2023-04-12 13:55:24,571 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/492a5a77fa3d400bbd80e56f5c7e2c90, entries=150, sequenceid=271, filesize=11.9 K 2023-04-12 13:55:24,571 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/b4596b8fc36f43138b7522ceeeb69fb3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/b4596b8fc36f43138b7522ceeeb69fb3 2023-04-12 13:55:24,574 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/b4596b8fc36f43138b7522ceeeb69fb3, entries=150, sequenceid=271, filesize=11.9 K 2023-04-12 13:55:24,575 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7eb722b369ab23b3fdff931ce8c3cac2 in 889ms, sequenceid=271, compaction requested=false 2023-04-12 13:55:24,575 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:24,575 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:24,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:24,579 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,580 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:24,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7607d7ca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:24,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:24,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,598 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,598 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:24,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:24,598 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:24,598 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:24,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:24,606 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:24,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:24,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:24,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:24,607 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:24,608 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,608 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,610 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,612 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:24,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,616 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:24,621 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:24,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:24,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:24,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:24,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:24,622 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:24,622 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:24,722 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:24,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:24,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:24,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:24,729 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:24,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:24,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:24,729 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:24,730 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,730 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:24,731 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:24,731 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:24,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:24,732 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:24,732 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:24,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:24,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:24,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:24,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:24,740 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:24,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:24,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:24,740 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:55:24,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:24,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:24,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:24,741 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:24,741 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:24,741 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:24,772 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d79f65281d9245169fe793e24beea2fd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d79f65281d9245169fe793e24beea2fd 2023-04-12 13:55:24,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:24,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:25,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307784991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307784992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307785000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307785000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307785101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307785101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307785110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307785110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,182 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/995dbaa5067f4297842171bac74d88ff 2023-04-12 13:55:25,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307785306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307785308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307785314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307785315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,595 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/1fd9d5dcc0b94133a5f9c37985a5535d 2023-04-12 13:55:25,604 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/2484a60ea24845889ad67ae3c914507a 2023-04-12 13:55:25,608 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/995dbaa5067f4297842171bac74d88ff as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/995dbaa5067f4297842171bac74d88ff 2023-04-12 13:55:25,611 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/995dbaa5067f4297842171bac74d88ff, entries=150, sequenceid=291, filesize=25.4 K 2023-04-12 13:55:25,612 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/1fd9d5dcc0b94133a5f9c37985a5535d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/1fd9d5dcc0b94133a5f9c37985a5535d 2023-04-12 13:55:25,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307785612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,615 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/1fd9d5dcc0b94133a5f9c37985a5535d, entries=150, sequenceid=291, filesize=11.9 K 2023-04-12 13:55:25,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307785613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,616 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/2484a60ea24845889ad67ae3c914507a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/2484a60ea24845889ad67ae3c914507a 2023-04-12 13:55:25,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307785619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307785621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,628 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/2484a60ea24845889ad67ae3c914507a, entries=150, sequenceid=291, filesize=11.9 K 2023-04-12 13:55:25,629 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 7eb722b369ab23b3fdff931ce8c3cac2 in 889ms, sequenceid=291, compaction requested=true 2023-04-12 13:55:25,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:25,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:25,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:25,631 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,631 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:25,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eeaa6c2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:25,633 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:25,633 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,635 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,635 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:25,635 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:25,635 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:25,635 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,635 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:25,637 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:25,637 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:25,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,637 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:25,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:25,642 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:25,642 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:25,642 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:25,642 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:25,642 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:25,645 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,645 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,645 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,645 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:25,646 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,653 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,653 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:25,653 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:25,653 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:25,653 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:25,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:25,653 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:25,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:25,654 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,654 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:25,654 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:25,653 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:25,654 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:25,654 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:25,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:25,754 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:25,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:25,758 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:25,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:25,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:25,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:25,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:25,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:25,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:25,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:25,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:25,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:25,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:25,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:25,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:25,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:25,762 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:25,762 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:25,762 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:25,762 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:55:25,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:25,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:25,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:25,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:25,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:25,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:26,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:26,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307786142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307786143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307786143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307786144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,193 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412f6c2367e85ed487e88a5c6bb188c4b76 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f6c2367e85ed487e88a5c6bb188c4b76 2023-04-12 13:55:26,207 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/cec57b0e69aa44a98768aeffe9d1079c 2023-04-12 13:55:26,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307786250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307786255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307786256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307786256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307786455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307786460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307786461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307786463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,620 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/3491af705aeb43a0af7b0237e066ab33 2023-04-12 13:55:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307786763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307786767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307786768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:26,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:26,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307786769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,030 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/d9d4fe930523467ca5fc2033fce6a008 2023-04-12 13:55:27,033 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/cec57b0e69aa44a98768aeffe9d1079c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/cec57b0e69aa44a98768aeffe9d1079c 2023-04-12 13:55:27,037 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/cec57b0e69aa44a98768aeffe9d1079c, entries=150, sequenceid=306, filesize=25.4 K 2023-04-12 13:55:27,037 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/3491af705aeb43a0af7b0237e066ab33 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/3491af705aeb43a0af7b0237e066ab33 2023-04-12 13:55:27,041 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/3491af705aeb43a0af7b0237e066ab33, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:55:27,042 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/d9d4fe930523467ca5fc2033fce6a008 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/d9d4fe930523467ca5fc2033fce6a008 2023-04-12 13:55:27,045 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/d9d4fe930523467ca5fc2033fce6a008, entries=150, sequenceid=306, filesize=11.9 K 2023-04-12 13:55:27,045 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1283ms, sequenceid=306, compaction requested=true 2023-04-12 13:55:27,045 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:27,045 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:27,047 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:27,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:27,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:27,048 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:27,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:27,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4734e727[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:27,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:27,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:27,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:27,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:27,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,056 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:27,057 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:27,059 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:27,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:27,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:27,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:27,060 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:27,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,062 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:27,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,063 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:27,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:27,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:27,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:27,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:27,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:27,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:27,068 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:27,068 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:27,068 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:27,169 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:27,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:27,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:27,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:27,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:27,179 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:27,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:27,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:27,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:27,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:27,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,184 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:27,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:27,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:27,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:27,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:27,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:27,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:27,225 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:27,225 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:27,226 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:55:27,226 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:27,226 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:27,226 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:27,226 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:27,226 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:27,226 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:27,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:27,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307787292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307787293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307787297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307787302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307787403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307787406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307787408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307787413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307787612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307787618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307787619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307787621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,674 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121b26febdc3534715a4d7c88270b2d464 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121b26febdc3534715a4d7c88270b2d464 2023-04-12 13:55:27,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307787923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307787924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307787926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:27,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:27,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307787926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,079 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=323, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/2e3fe5e5aeaf40828ce930f8b48c13f9 2023-04-12 13:55:28,089 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/e1343d9df89a4a25aed4828d912aea24 2023-04-12 13:55:28,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307788432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307788433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307788433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307788436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,504 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/4bd2c48dbeb74d348849b0f9f174a7fd 2023-04-12 13:55:28,508 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/2e3fe5e5aeaf40828ce930f8b48c13f9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/2e3fe5e5aeaf40828ce930f8b48c13f9 2023-04-12 13:55:28,511 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/2e3fe5e5aeaf40828ce930f8b48c13f9, entries=150, sequenceid=323, filesize=25.4 K 2023-04-12 13:55:28,512 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/e1343d9df89a4a25aed4828d912aea24 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e1343d9df89a4a25aed4828d912aea24 2023-04-12 13:55:28,515 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e1343d9df89a4a25aed4828d912aea24, entries=150, sequenceid=323, filesize=11.9 K 2023-04-12 13:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/4bd2c48dbeb74d348849b0f9f174a7fd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4bd2c48dbeb74d348849b0f9f174a7fd 2023-04-12 13:55:28,519 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4bd2c48dbeb74d348849b0f9f174a7fd, entries=150, sequenceid=323, filesize=11.9 K 2023-04-12 13:55:28,520 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1294ms, sequenceid=323, compaction requested=true 2023-04-12 13:55:28,520 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:28,520 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,522 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:28,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f040692[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:28,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:28,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,525 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:28,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,526 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:28,527 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:28,527 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:28,530 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:28,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:28,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:28,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:28,530 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:28,532 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,532 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,532 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:28,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:28,532 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:28,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:28,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:28,536 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:28,536 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:28,536 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:28,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:28,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:28,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:28,636 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:28,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:28,639 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:28,639 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:28,639 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:28,639 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:28,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:28,641 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:28,641 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:28,641 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:28,641 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:28,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:28,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:28,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:28,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:28,642 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:28,642 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:28,643 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:28,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:29,058 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041233e49a9925b749d987fd2da4a4eec651 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041233e49a9925b749d987fd2da4a4eec651 2023-04-12 13:55:29,067 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=338, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/12abb41167d0429f8bce3f8cd42da1c8 2023-04-12 13:55:29,087 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/a6b3e8e92d804d898f2b19cfed7e5ee1 2023-04-12 13:55:29,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:29,449 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:29,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307789468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307789473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307789474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307789476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,502 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/6938a8cc98994c2e9978d3d68733d6ab 2023-04-12 13:55:29,506 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/12abb41167d0429f8bce3f8cd42da1c8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/12abb41167d0429f8bce3f8cd42da1c8 2023-04-12 13:55:29,509 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/12abb41167d0429f8bce3f8cd42da1c8, entries=150, sequenceid=338, filesize=25.4 K 2023-04-12 13:55:29,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/a6b3e8e92d804d898f2b19cfed7e5ee1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/a6b3e8e92d804d898f2b19cfed7e5ee1 2023-04-12 13:55:29,513 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/a6b3e8e92d804d898f2b19cfed7e5ee1, entries=150, sequenceid=338, filesize=11.9 K 2023-04-12 13:55:29,514 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/6938a8cc98994c2e9978d3d68733d6ab as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6938a8cc98994c2e9978d3d68733d6ab 2023-04-12 13:55:29,517 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6938a8cc98994c2e9978d3d68733d6ab, entries=150, sequenceid=338, filesize=11.9 K 2023-04-12 13:55:29,518 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 7eb722b369ab23b3fdff931ce8c3cac2 in 875ms, sequenceid=338, compaction requested=true 2023-04-12 13:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:29,518 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:29,520 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,521 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f5333f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:29,522 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:29,522 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,523 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:29,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:29,523 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:29,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,523 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:29,525 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:29,525 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,525 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,525 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:29,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:29,528 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:29,528 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:29,528 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:29,528 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:29,528 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:29,529 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,529 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,529 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,531 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,542 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:29,543 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:29,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,543 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:29,543 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:29,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:29,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:55:29,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:29,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:29,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:29,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:29,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:29,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:29,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307789608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307789608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307789610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307789614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,643 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:29,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:29,647 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:29,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:29,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:29,647 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:29,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,648 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,650 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,650 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:29,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:29,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:29,652 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,652 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:29,652 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:29,652 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:29,652 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:29,654 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:29,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca25bb5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:29,656 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:29,656 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,657 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:29,657 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,657 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:29,659 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:29,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,659 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:29,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:29,661 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:29,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:29,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:29,661 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:29,661 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:29,663 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,663 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,663 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,664 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,665 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:29,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,666 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,667 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:29,667 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:29,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:29,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307789719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307789719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307789720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307789722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,767 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:29,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:29,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:29,771 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:29,771 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:29,771 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:29,771 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:29,772 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,772 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,773 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,773 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:29,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:29,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:29,775 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:29,775 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:29,775 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:29,775 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:29,775 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:29,775 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:29,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:29,776 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:29,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,778 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:29,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccf9ad9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:29,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:29,780 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,782 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:29,782 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,782 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:29,784 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:29,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,784 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:29,786 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:29,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:29,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:29,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:29,786 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:29,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,788 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,789 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,792 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:29,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:29,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,794 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:29,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:29,794 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:29,894 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:29,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:29,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:29,897 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:29,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:29,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:29,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:29,898 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,898 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,900 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,900 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,900 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:29,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:29,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,903 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3802344[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:29,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:29,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,907 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:29,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:29,907 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,907 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:29,907 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,909 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:29,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:29,911 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:29,911 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:29,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:29,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:29,912 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:29,914 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,914 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,915 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:29,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:29,915 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,916 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:29,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:29,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:29,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:29,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:29,919 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:29,919 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:29,919 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:29,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:29,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307789926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307789926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307789927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:29,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307789930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:29,997 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304129724b38a76b849f4a27515414ea97600 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129724b38a76b849f4a27515414ea97600 2023-04-12 13:55:30,019 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,024 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,034 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,035 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,036 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,038 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d5294c0[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,043 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,044 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,045 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,045 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,045 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,046 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,049 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,049 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,050 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,052 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,052 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,052 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,058 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,059 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,059 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,059 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,159 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,163 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,166 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,166 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,167 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,167 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,168 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,169 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33a31f2c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,172 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,174 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,174 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,174 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,176 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,177 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,177 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,178 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,178 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,179 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,186 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:30,187 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,187 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307790230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307790231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307790234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307790239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,287 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,291 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,292 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,292 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,294 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,294 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,294 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,294 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,295 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333a42d3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,306 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,306 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,306 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,313 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,313 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,406 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/d9a826f50643485d9f62b497f789ac37 2023-04-12 13:55:30,413 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,431 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,440 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,445 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,445 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3122ddf3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,447 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,447 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,450 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,451 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,451 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,453 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,453 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,453 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,453 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,455 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,456 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,456 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,457 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,459 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,459 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,462 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,464 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,464 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,464 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,464 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,464 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,465 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,465 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,565 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,568 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,572 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,573 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,574 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,574 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,574 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,575 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,576 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,576 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187fab83[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,578 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,578 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,578 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,580 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,580 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,582 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,582 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,582 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,582 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,582 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,587 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,589 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,589 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,590 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,690 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,693 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,695 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,699 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,699 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338e8ddf[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,701 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,701 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,704 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,704 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,707 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,707 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,712 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,712 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,712 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,713 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,712 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,713 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,713 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307790748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307790748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307790750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:30,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307790750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,813 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,817 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,817 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,817 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,817 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,817 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,824 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,824 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,825 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,825 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,825 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,826 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,826 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,827 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52436da7[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,829 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/382992ea3316499a8fa59cf5abd44cc4 2023-04-12 13:55:30,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,829 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,836 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,839 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,839 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,839 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,840 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,841 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,844 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,845 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,846 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,846 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,946 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:30,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:30,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:30,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:30,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:30,950 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:30,950 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,950 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:30,950 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:30,951 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,951 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,952 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,953 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,953 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,953 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:30,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:30,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:30,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:30,955 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:30,964 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,964 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:30,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7529ef59[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:30,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:30,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,967 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,967 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:30,971 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:30,972 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:30,972 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:30,972 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:30,972 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:30,975 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:30,976 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,976 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:30,976 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:30,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:30,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:30,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:30,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:30,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:30,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:30,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:30,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:30,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:30,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:30,982 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:30,982 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:30,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:31,082 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:31,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:31,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:31,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:31,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:31,085 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:31,085 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:31,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:31,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:31,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,087 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:31,087 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:31,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:31,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:31,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:31,089 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,089 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:31,089 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:31,089 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:31,092 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ef61cc[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:31,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:31,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,096 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:31,096 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:31,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,099 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:31,101 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:31,101 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:31,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:31,104 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:31,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:31,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:31,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:31,104 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:31,106 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,106 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:31,106 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:31,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:31,107 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,108 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,110 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,110 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:31,110 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:31,110 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:31,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:31,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,111 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:31,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:31,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:31,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:31,111 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:31,112 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:31,212 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:31,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:31,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:31,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:31,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:31,217 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:31,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:31,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:31,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:31,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,225 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:31,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:31,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,226 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60854 deadline: 1681307791216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:31,228 DEBUG [Thread-2236] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:55:31,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:31,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:31,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:31,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:31,232 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:31,232 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,236 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c8f67b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:31,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:31,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:31,242 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,242 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:31,242 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:31,248 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:31,248 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,248 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:31,248 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:31,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:31,250 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:31,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:31,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:31,250 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:31,250 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:31,257 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,257 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,257 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:31,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:31,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:31,258 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:31,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:31,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:31,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:31,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:31,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,264 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:31,264 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:31,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:31,264 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:31,264 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:31,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:31,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/4c3ced06dd4344edbb9b36a52557690c 2023-04-12 13:55:31,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/d9a826f50643485d9f62b497f789ac37 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/d9a826f50643485d9f62b497f789ac37 2023-04-12 13:55:31,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/d9a826f50643485d9f62b497f789ac37, entries=200, sequenceid=356, filesize=32.3 K 2023-04-12 13:55:31,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/382992ea3316499a8fa59cf5abd44cc4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/382992ea3316499a8fa59cf5abd44cc4 2023-04-12 13:55:31,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/382992ea3316499a8fa59cf5abd44cc4, entries=150, sequenceid=356, filesize=11.9 K 2023-04-12 13:55:31,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/4c3ced06dd4344edbb9b36a52557690c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c3ced06dd4344edbb9b36a52557690c 2023-04-12 13:55:31,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c3ced06dd4344edbb9b36a52557690c, entries=150, sequenceid=356, filesize=11.9 K 2023-04-12 13:55:31,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1704ms, sequenceid=356, compaction requested=true 2023-04-12 13:55:31,288 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:31,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:31,288 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:31,288 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:31,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:31,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:31,290 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 189792 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:31,290 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/A is initiating minor compaction (all files) 2023-04-12 13:55:31,290 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/A in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,290 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4f771522c9cd431e9133b56e1f017592, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6ff22e8312144150a5e5595a80eac4c6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/995dbaa5067f4297842171bac74d88ff, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/cec57b0e69aa44a98768aeffe9d1079c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/2e3fe5e5aeaf40828ce930f8b48c13f9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/12abb41167d0429f8bce3f8cd42da1c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/d9a826f50643485d9f62b497f789ac37] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=185.3 K 2023-04-12 13:55:31,291 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85732 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:31,291 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/B is initiating minor compaction (all files) 2023-04-12 13:55:31,291 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/B in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,291 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/88df68f32b9a4153a17277367dee6d7e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/492a5a77fa3d400bbd80e56f5c7e2c90, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/1fd9d5dcc0b94133a5f9c37985a5535d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/3491af705aeb43a0af7b0237e066ab33, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e1343d9df89a4a25aed4828d912aea24, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/a6b3e8e92d804d898f2b19cfed7e5ee1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/382992ea3316499a8fa59cf5abd44cc4] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=83.7 K 2023-04-12 13:55:31,291 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4f771522c9cd431e9133b56e1f017592, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681307722808 2023-04-12 13:55:31,291 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 88df68f32b9a4153a17277367dee6d7e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681307722808 2023-04-12 13:55:31,292 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6ff22e8312144150a5e5595a80eac4c6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681307723168 2023-04-12 13:55:31,292 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 492a5a77fa3d400bbd80e56f5c7e2c90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681307723168 2023-04-12 13:55:31,292 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 995dbaa5067f4297842171bac74d88ff, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681307723845 2023-04-12 13:55:31,292 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1fd9d5dcc0b94133a5f9c37985a5535d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681307723845 2023-04-12 13:55:31,292 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting cec57b0e69aa44a98768aeffe9d1079c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681307724989 2023-04-12 13:55:31,292 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3491af705aeb43a0af7b0237e066ab33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681307724989 2023-04-12 13:55:31,293 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2e3fe5e5aeaf40828ce930f8b48c13f9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681307726133 2023-04-12 13:55:31,293 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e1343d9df89a4a25aed4828d912aea24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681307726133 2023-04-12 13:55:31,293 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 12abb41167d0429f8bce3f8cd42da1c8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681307727290 2023-04-12 13:55:31,293 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a6b3e8e92d804d898f2b19cfed7e5ee1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681307727290 2023-04-12 13:55:31,293 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d9a826f50643485d9f62b497f789ac37, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681307729466 2023-04-12 13:55:31,293 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 382992ea3316499a8fa59cf5abd44cc4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681307729472 2023-04-12 13:55:31,309 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#B#compaction#375 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:31,315 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#A#compaction#374 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:31,364 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:31,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:31,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:31,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:31,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:31,372 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:31,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:31,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:31,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:31,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,373 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,374 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:31,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:31,374 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:31,375 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,375 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:31,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:31,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:31,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:31,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:31,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:31,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:31,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:31,376 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:31,376 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,376 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:31,376 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:55:31,376 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:31,377 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:31,377 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:31,377 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:31,377 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:31,377 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:31,721 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/335ee5f8a69a46628eab9581ce2c6377 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/335ee5f8a69a46628eab9581ce2c6377 2023-04-12 13:55:31,733 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/B of 7eb722b369ab23b3fdff931ce8c3cac2 into 335ee5f8a69a46628eab9581ce2c6377(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:31,733 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:31,733 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/B, priority=9, startTime=1681307731288; duration=0sec 2023-04-12 13:55:31,733 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:31,733 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:31,737 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85732 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:31,737 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): 7eb722b369ab23b3fdff931ce8c3cac2/C is initiating minor compaction (all files) 2023-04-12 13:55:31,737 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 7eb722b369ab23b3fdff931ce8c3cac2/C in TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:31,737 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/14515f8dc8e24e1086c4a81ebc902058, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/b4596b8fc36f43138b7522ceeeb69fb3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/2484a60ea24845889ad67ae3c914507a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/d9d4fe930523467ca5fc2033fce6a008, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4bd2c48dbeb74d348849b0f9f174a7fd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6938a8cc98994c2e9978d3d68733d6ab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c3ced06dd4344edbb9b36a52557690c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp, totalSize=83.7 K 2023-04-12 13:55:31,738 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 14515f8dc8e24e1086c4a81ebc902058, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681307722808 2023-04-12 13:55:31,738 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b4596b8fc36f43138b7522ceeeb69fb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681307723168 2023-04-12 13:55:31,738 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2484a60ea24845889ad67ae3c914507a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681307723845 2023-04-12 13:55:31,739 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d9d4fe930523467ca5fc2033fce6a008, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681307724989 2023-04-12 13:55:31,739 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4bd2c48dbeb74d348849b0f9f174a7fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681307726133 2023-04-12 13:55:31,739 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6938a8cc98994c2e9978d3d68733d6ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681307727290 2023-04-12 13:55:31,739 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4c3ced06dd4344edbb9b36a52557690c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681307729472 2023-04-12 13:55:31,742 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/f863f137b82d4287ae90b871350a626e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f863f137b82d4287ae90b871350a626e 2023-04-12 13:55:31,746 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/A of 7eb722b369ab23b3fdff931ce8c3cac2 into f863f137b82d4287ae90b871350a626e(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:31,746 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:31,746 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/A, priority=9, startTime=1681307731288; duration=0sec 2023-04-12 13:55:31,746 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:31,754 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): 7eb722b369ab23b3fdff931ce8c3cac2#C#compaction#377 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:31,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:31,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. as already flushing 2023-04-12 13:55:31,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307791786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307791787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307791792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307791793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,815 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304123c7fdd7c57f54da4ba0effa54cdf5bc0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123c7fdd7c57f54da4ba0effa54cdf5bc0 2023-04-12 13:55:31,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307791894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307791894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307791895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:31,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:31,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307791901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307792103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307792104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307792104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307792105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,163 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/9e3173139e0e475096944e6b0e3dc87d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/9e3173139e0e475096944e6b0e3dc87d 2023-04-12 13:55:32,167 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 7eb722b369ab23b3fdff931ce8c3cac2/C of 7eb722b369ab23b3fdff931ce8c3cac2 into 9e3173139e0e475096944e6b0e3dc87d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:32,167 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:32,167 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2., storeName=7eb722b369ab23b3fdff931ce8c3cac2/C, priority=9, startTime=1681307731288; duration=0sec 2023-04-12 13:55:32,167 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:32,221 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/da9ae552c4f3422a889432e0ffc6aba0 2023-04-12 13:55:32,234 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/ccf9113978194fb3b1e33a4b0f5c875c 2023-04-12 13:55:32,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60790 deadline: 1681307792408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60798 deadline: 1681307792409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60830 deadline: 1681307792409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:32,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60826 deadline: 1681307792411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,646 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/6febb1785acc442aada6845e10f06320 2023-04-12 13:55:32,650 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/da9ae552c4f3422a889432e0ffc6aba0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/da9ae552c4f3422a889432e0ffc6aba0 2023-04-12 13:55:32,653 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/da9ae552c4f3422a889432e0ffc6aba0, entries=150, sequenceid=370, filesize=25.4 K 2023-04-12 13:55:32,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/ccf9113978194fb3b1e33a4b0f5c875c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ccf9113978194fb3b1e33a4b0f5c875c 2023-04-12 13:55:32,657 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ccf9113978194fb3b1e33a4b0f5c875c, entries=150, sequenceid=370, filesize=11.9 K 2023-04-12 13:55:32,658 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/6febb1785acc442aada6845e10f06320 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6febb1785acc442aada6845e10f06320 2023-04-12 13:55:32,661 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6febb1785acc442aada6845e10f06320, entries=150, sequenceid=370, filesize=11.9 K 2023-04-12 13:55:32,661 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 7eb722b369ab23b3fdff931ce8c3cac2 in 1285ms, sequenceid=370, compaction requested=false 2023-04-12 13:55:32,661 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:32,662 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:32,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:32,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:32,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:32,662 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:32,664 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:32,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,664 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:32,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eaf9770[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:32,665 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:32,665 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,666 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,667 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:32,667 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:32,667 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:32,667 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,667 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:32,668 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:32,670 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:32,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:32,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:32,670 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:32,670 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:32,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:32,672 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:32,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:32,673 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,674 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:32,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:32,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:32,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:32,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:32,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:32,679 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:32,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:32,679 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:32,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:32,781 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:32,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:32,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:32,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:32,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:32,783 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:32,783 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:32,784 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:32,784 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:32,784 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,784 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,785 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:32,785 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:32,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:32,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,786 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:32,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:32,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:32,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:32,788 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:32,788 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:32,788 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. started... 2023-04-12 13:55:32,789 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-12 13:55:32,789 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:32,789 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:32,789 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:32,789 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:32,789 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:32,789 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:32,818 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a93de667bd5b40d7805d6ac87348f24f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a93de667bd5b40d7805d6ac87348f24f 2023-04-12 13:55:32,844 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/ae000806cac54d19ab646dd39032699e 2023-04-12 13:55:32,858 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/4b9dd100637448b995a40c0f5d11f62f 2023-04-12 13:55:32,882 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/611ff981da54416cbb741ad7ce284114 2023-04-12 13:55:32,886 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/ae000806cac54d19ab646dd39032699e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/ae000806cac54d19ab646dd39032699e 2023-04-12 13:55:32,890 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/ae000806cac54d19ab646dd39032699e, entries=150, sequenceid=391, filesize=25.4 K 2023-04-12 13:55:32,891 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/4b9dd100637448b995a40c0f5d11f62f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b9dd100637448b995a40c0f5d11f62f 2023-04-12 13:55:32,894 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b9dd100637448b995a40c0f5d11f62f, entries=150, sequenceid=391, filesize=11.9 K 2023-04-12 13:55:32,894 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/611ff981da54416cbb741ad7ce284114 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/611ff981da54416cbb741ad7ce284114 2023-04-12 13:55:32,901 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/611ff981da54416cbb741ad7ce284114, entries=150, sequenceid=391, filesize=11.9 K 2023-04-12 13:55:32,902 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 7eb722b369ab23b3fdff931ce8c3cac2 in 113ms, sequenceid=391, compaction requested=true 2023-04-12 13:55:32,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:32,902 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:32,902 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:32,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:32,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:32,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2446f0db[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:32,907 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:32,907 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:32,908 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:32,909 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,909 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:32,911 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:32,911 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:32,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:32,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:32,915 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:32,915 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:32,915 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:32,915 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:32,916 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:32,917 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,917 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:32,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:32,918 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,920 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:32,922 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:32,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:32,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:32,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:32,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:32,922 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:32,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:32,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:32,923 DEBUG [Thread-2240] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:32,923 DEBUG [Thread-2240] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:32,923 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:32,972 DEBUG [Thread-2247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:58527 2023-04-12 13:55:32,972 DEBUG [Thread-2247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:32,973 DEBUG [Thread-2245] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:58527 2023-04-12 13:55:32,973 DEBUG [Thread-2245] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:32,974 DEBUG [Thread-2238] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:58527 2023-04-12 13:55:32,974 DEBUG [Thread-2238] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:32,974 DEBUG [Thread-2230] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:58527 2023-04-12 13:55:32,974 DEBUG [Thread-2230] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:32,975 DEBUG [Thread-2234] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:58527 2023-04-12 13:55:32,976 DEBUG [Thread-2234] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:32,978 DEBUG [Thread-2232] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:58527 2023-04-12 13:55:32,978 DEBUG [Thread-2232] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:33,007 DEBUG [Thread-2243] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:58527 2023-04-12 13:55:33,007 DEBUG [Thread-2243] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:33,008 DEBUG [Thread-2249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:58527 2023-04-12 13:55:33,008 DEBUG [Thread-2249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:33,023 DEBUG [Thread-2240] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:33,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:33,050 DEBUG [Thread-2241] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:58527 2023-04-12 13:55:33,050 DEBUG [Thread-2241] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:39,146 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 13:55:39,180 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/50a2545fda2f4d6cbe530a075a73a2de, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4a0cc2b8844241a69a3e583ce3517c65, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6b14d410d0ff46e4b4bbc1873776332f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/34e388254ed94cbb86c77e4dd688cd34, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a228fde2e1f44d13a74ebe8cfbfed487, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/bf31869d71ed43ce9daeda7ff8dd339f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/dc1ac3967a084154876e0361a9c7de9f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/675308ab8cd14a45a35fae02497cec79, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/b41993703c814b17a8ab7a61a8b54cb6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/159afe661c254e5b982d9215e3412f96, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f416ff4fa4cd4158a9ef1b11df7e0496, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/910a3790d9074ee2931bc45fc4e14a0c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/525b11a9c4084d2f8bd64ebdd30c988c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/5fb2844bb2cb4b0a8bfa0b6268ec3819, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a7a88d8f996a4558abc75293cfc0a192, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/04c368c60c104c90a921201de48f8d8f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/311e558ee7ee40c596023ef8aeb962be, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4f771522c9cd431e9133b56e1f017592, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6ff22e8312144150a5e5595a80eac4c6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/995dbaa5067f4297842171bac74d88ff, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/cec57b0e69aa44a98768aeffe9d1079c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/2e3fe5e5aeaf40828ce930f8b48c13f9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/12abb41167d0429f8bce3f8cd42da1c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/d9a826f50643485d9f62b497f789ac37] to archive 2023-04-12 13:55:39,181 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:55:39,183 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/50a2545fda2f4d6cbe530a075a73a2de to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/50a2545fda2f4d6cbe530a075a73a2de 2023-04-12 13:55:39,186 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4a0cc2b8844241a69a3e583ce3517c65 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4a0cc2b8844241a69a3e583ce3517c65 2023-04-12 13:55:39,188 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6b14d410d0ff46e4b4bbc1873776332f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6b14d410d0ff46e4b4bbc1873776332f 2023-04-12 13:55:39,189 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/34e388254ed94cbb86c77e4dd688cd34 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/34e388254ed94cbb86c77e4dd688cd34 2023-04-12 13:55:39,190 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a228fde2e1f44d13a74ebe8cfbfed487 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a228fde2e1f44d13a74ebe8cfbfed487 2023-04-12 13:55:39,192 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/bf31869d71ed43ce9daeda7ff8dd339f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/bf31869d71ed43ce9daeda7ff8dd339f 2023-04-12 13:55:39,193 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/dc1ac3967a084154876e0361a9c7de9f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/dc1ac3967a084154876e0361a9c7de9f 2023-04-12 13:55:39,195 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/675308ab8cd14a45a35fae02497cec79 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/675308ab8cd14a45a35fae02497cec79 2023-04-12 13:55:39,196 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/b41993703c814b17a8ab7a61a8b54cb6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/b41993703c814b17a8ab7a61a8b54cb6 2023-04-12 13:55:39,198 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/159afe661c254e5b982d9215e3412f96 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/159afe661c254e5b982d9215e3412f96 2023-04-12 13:55:39,199 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f416ff4fa4cd4158a9ef1b11df7e0496 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f416ff4fa4cd4158a9ef1b11df7e0496 2023-04-12 13:55:39,200 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/910a3790d9074ee2931bc45fc4e14a0c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/910a3790d9074ee2931bc45fc4e14a0c 2023-04-12 13:55:39,201 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/525b11a9c4084d2f8bd64ebdd30c988c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/525b11a9c4084d2f8bd64ebdd30c988c 2023-04-12 13:55:39,203 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/5fb2844bb2cb4b0a8bfa0b6268ec3819 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/5fb2844bb2cb4b0a8bfa0b6268ec3819 2023-04-12 13:55:39,204 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a7a88d8f996a4558abc75293cfc0a192 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/a7a88d8f996a4558abc75293cfc0a192 2023-04-12 13:55:39,205 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/04c368c60c104c90a921201de48f8d8f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/04c368c60c104c90a921201de48f8d8f 2023-04-12 13:55:39,207 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/311e558ee7ee40c596023ef8aeb962be to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/311e558ee7ee40c596023ef8aeb962be 2023-04-12 13:55:39,212 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4f771522c9cd431e9133b56e1f017592 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/4f771522c9cd431e9133b56e1f017592 2023-04-12 13:55:39,213 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6ff22e8312144150a5e5595a80eac4c6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/6ff22e8312144150a5e5595a80eac4c6 2023-04-12 13:55:39,215 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/995dbaa5067f4297842171bac74d88ff to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/995dbaa5067f4297842171bac74d88ff 2023-04-12 13:55:39,216 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/cec57b0e69aa44a98768aeffe9d1079c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/cec57b0e69aa44a98768aeffe9d1079c 2023-04-12 13:55:39,217 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/2e3fe5e5aeaf40828ce930f8b48c13f9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/2e3fe5e5aeaf40828ce930f8b48c13f9 2023-04-12 13:55:39,219 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/12abb41167d0429f8bce3f8cd42da1c8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/12abb41167d0429f8bce3f8cd42da1c8 2023-04-12 13:55:39,220 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/d9a826f50643485d9f62b497f789ac37 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/d9a826f50643485d9f62b497f789ac37 2023-04-12 13:55:39,224 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/63e369b188da4c538468c589f952ba27, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/50721cbe55ca44d0bf92ee40f5dc6400, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/f66de41b98d64479b28829304d1f98f9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/85892f549be6484ebf92472a962ca452, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/21f0449168c04306ba7619f098e6b149, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8fe2e3f8baa34ab0be520b70707e2377, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/553d8d4987f14594b18e129a44b79989, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b959efb851244389062b01c19be63ee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e71ed46191ac4169ab64a125fe328558, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6ea24902a19f46f18671affc7696e140, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/74a854b1e752452fb9df613e8bfcd765, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/93472f11491a48e7b22e5ae28b57fb04, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6982987c663043a98bb35ab6dfd01a40, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ad5ca44173bc4bb1ba2321547e3d84b0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8639b86721fb46a0aa4a69e72bc221f8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/39840b19e73f4ed4a66c8bd20792205b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/88df68f32b9a4153a17277367dee6d7e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/d3337e3844414cd68e745c38bbeee900, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/492a5a77fa3d400bbd80e56f5c7e2c90, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/1fd9d5dcc0b94133a5f9c37985a5535d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/3491af705aeb43a0af7b0237e066ab33, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e1343d9df89a4a25aed4828d912aea24, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/a6b3e8e92d804d898f2b19cfed7e5ee1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/382992ea3316499a8fa59cf5abd44cc4] to archive 2023-04-12 13:55:39,225 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:55:39,226 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/63e369b188da4c538468c589f952ba27 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/63e369b188da4c538468c589f952ba27 2023-04-12 13:55:39,227 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/50721cbe55ca44d0bf92ee40f5dc6400 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/50721cbe55ca44d0bf92ee40f5dc6400 2023-04-12 13:55:39,228 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/f66de41b98d64479b28829304d1f98f9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/f66de41b98d64479b28829304d1f98f9 2023-04-12 13:55:39,229 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/85892f549be6484ebf92472a962ca452 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/85892f549be6484ebf92472a962ca452 2023-04-12 13:55:39,230 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/21f0449168c04306ba7619f098e6b149 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/21f0449168c04306ba7619f098e6b149 2023-04-12 13:55:39,231 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8fe2e3f8baa34ab0be520b70707e2377 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8fe2e3f8baa34ab0be520b70707e2377 2023-04-12 13:55:39,232 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/553d8d4987f14594b18e129a44b79989 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/553d8d4987f14594b18e129a44b79989 2023-04-12 13:55:39,233 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b959efb851244389062b01c19be63ee to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b959efb851244389062b01c19be63ee 2023-04-12 13:55:39,234 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e71ed46191ac4169ab64a125fe328558 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e71ed46191ac4169ab64a125fe328558 2023-04-12 13:55:39,235 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6ea24902a19f46f18671affc7696e140 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6ea24902a19f46f18671affc7696e140 2023-04-12 13:55:39,236 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/74a854b1e752452fb9df613e8bfcd765 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/74a854b1e752452fb9df613e8bfcd765 2023-04-12 13:55:39,237 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/93472f11491a48e7b22e5ae28b57fb04 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/93472f11491a48e7b22e5ae28b57fb04 2023-04-12 13:55:39,238 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6982987c663043a98bb35ab6dfd01a40 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/6982987c663043a98bb35ab6dfd01a40 2023-04-12 13:55:39,239 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ad5ca44173bc4bb1ba2321547e3d84b0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ad5ca44173bc4bb1ba2321547e3d84b0 2023-04-12 13:55:39,240 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8639b86721fb46a0aa4a69e72bc221f8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/8639b86721fb46a0aa4a69e72bc221f8 2023-04-12 13:55:39,241 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/39840b19e73f4ed4a66c8bd20792205b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/39840b19e73f4ed4a66c8bd20792205b 2023-04-12 13:55:39,242 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/88df68f32b9a4153a17277367dee6d7e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/88df68f32b9a4153a17277367dee6d7e 2023-04-12 13:55:39,243 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/d3337e3844414cd68e745c38bbeee900 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/d3337e3844414cd68e745c38bbeee900 2023-04-12 13:55:39,244 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/492a5a77fa3d400bbd80e56f5c7e2c90 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/492a5a77fa3d400bbd80e56f5c7e2c90 2023-04-12 13:55:39,245 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/1fd9d5dcc0b94133a5f9c37985a5535d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/1fd9d5dcc0b94133a5f9c37985a5535d 2023-04-12 13:55:39,246 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/3491af705aeb43a0af7b0237e066ab33 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/3491af705aeb43a0af7b0237e066ab33 2023-04-12 13:55:39,247 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e1343d9df89a4a25aed4828d912aea24 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/e1343d9df89a4a25aed4828d912aea24 2023-04-12 13:55:39,248 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/a6b3e8e92d804d898f2b19cfed7e5ee1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/a6b3e8e92d804d898f2b19cfed7e5ee1 2023-04-12 13:55:39,249 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/382992ea3316499a8fa59cf5abd44cc4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/382992ea3316499a8fa59cf5abd44cc4 2023-04-12 13:55:39,253 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/fa62044639bb408093c8ac6b90014436, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/78e98b57ba964f58b562b330eb10ed15, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/19d1dd58a90f49c0b9b686425a33fbd0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/42e99fe4715647b99b83bf190eec9db8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/10f9041a452e411c8c737ef63c1a001a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/80f517b7997d4f7bb13ef6ba15c0e174, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cad51693c04246a58db1bfab1936942c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cf4ccc21cfab4e9c86ad34dbe33774b4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/f3be966cf7d54208b6fda810a4cb94ee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/64d432f8ef1f4df5837dc01b65b8fe30, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ba6bc4729a124ef5a34eeec3d00ec57f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ddcb03097ca7435c84f508f22b09c1e1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/bcebb911f18241459bcedc2b040abdf1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/a57dbe39d2d745958f28aaaa1eb4ac7f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c8e8112387d4b4d88a616181f9c9c52, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6b8dac50f2cf43af8ab4b7618368d9e3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/14515f8dc8e24e1086c4a81ebc902058, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/b4596b8fc36f43138b7522ceeeb69fb3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/2484a60ea24845889ad67ae3c914507a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/d9d4fe930523467ca5fc2033fce6a008, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4bd2c48dbeb74d348849b0f9f174a7fd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6938a8cc98994c2e9978d3d68733d6ab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c3ced06dd4344edbb9b36a52557690c] to archive 2023-04-12 13:55:39,254 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:55:39,255 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/fa62044639bb408093c8ac6b90014436 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/fa62044639bb408093c8ac6b90014436 2023-04-12 13:55:39,256 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/78e98b57ba964f58b562b330eb10ed15 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/78e98b57ba964f58b562b330eb10ed15 2023-04-12 13:55:39,257 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/19d1dd58a90f49c0b9b686425a33fbd0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/19d1dd58a90f49c0b9b686425a33fbd0 2023-04-12 13:55:39,258 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/42e99fe4715647b99b83bf190eec9db8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/42e99fe4715647b99b83bf190eec9db8 2023-04-12 13:55:39,259 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/10f9041a452e411c8c737ef63c1a001a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/10f9041a452e411c8c737ef63c1a001a 2023-04-12 13:55:39,260 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/80f517b7997d4f7bb13ef6ba15c0e174 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/80f517b7997d4f7bb13ef6ba15c0e174 2023-04-12 13:55:39,261 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cad51693c04246a58db1bfab1936942c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cad51693c04246a58db1bfab1936942c 2023-04-12 13:55:39,262 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cf4ccc21cfab4e9c86ad34dbe33774b4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/cf4ccc21cfab4e9c86ad34dbe33774b4 2023-04-12 13:55:39,263 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/f3be966cf7d54208b6fda810a4cb94ee to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/f3be966cf7d54208b6fda810a4cb94ee 2023-04-12 13:55:39,264 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/64d432f8ef1f4df5837dc01b65b8fe30 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/64d432f8ef1f4df5837dc01b65b8fe30 2023-04-12 13:55:39,265 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ba6bc4729a124ef5a34eeec3d00ec57f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ba6bc4729a124ef5a34eeec3d00ec57f 2023-04-12 13:55:39,266 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ddcb03097ca7435c84f508f22b09c1e1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/ddcb03097ca7435c84f508f22b09c1e1 2023-04-12 13:55:39,267 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/bcebb911f18241459bcedc2b040abdf1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/bcebb911f18241459bcedc2b040abdf1 2023-04-12 13:55:39,268 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/a57dbe39d2d745958f28aaaa1eb4ac7f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/a57dbe39d2d745958f28aaaa1eb4ac7f 2023-04-12 13:55:39,269 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c8e8112387d4b4d88a616181f9c9c52 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c8e8112387d4b4d88a616181f9c9c52 2023-04-12 13:55:39,270 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6b8dac50f2cf43af8ab4b7618368d9e3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6b8dac50f2cf43af8ab4b7618368d9e3 2023-04-12 13:55:39,270 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/14515f8dc8e24e1086c4a81ebc902058 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/14515f8dc8e24e1086c4a81ebc902058 2023-04-12 13:55:39,271 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/1a24df7d9e5b4ffbaeaf932b5f07ecb4 2023-04-12 13:55:39,272 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/b4596b8fc36f43138b7522ceeeb69fb3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/b4596b8fc36f43138b7522ceeeb69fb3 2023-04-12 13:55:39,273 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/2484a60ea24845889ad67ae3c914507a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/2484a60ea24845889ad67ae3c914507a 2023-04-12 13:55:39,274 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/d9d4fe930523467ca5fc2033fce6a008 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/d9d4fe930523467ca5fc2033fce6a008 2023-04-12 13:55:39,275 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4bd2c48dbeb74d348849b0f9f174a7fd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4bd2c48dbeb74d348849b0f9f174a7fd 2023-04-12 13:55:39,275 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6938a8cc98994c2e9978d3d68733d6ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6938a8cc98994c2e9978d3d68733d6ab 2023-04-12 13:55:39,276 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c3ced06dd4344edbb9b36a52557690c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/4c3ced06dd4344edbb9b36a52557690c 2023-04-12 13:55:41,275 DEBUG [Thread-2236] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:58527 2023-04-12 13:55:41,275 DEBUG [Thread-2236] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:41,275 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 5 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1834 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5502 rows 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1829 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5487 rows 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1830 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5490 rows 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1836 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5494 rows 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1836 2023-04-12 13:55:41,276 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 5508 rows 2023-04-12 13:55:41,277 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:55:41,277 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:58527 2023-04-12 13:55:41,277 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:55:41,279 INFO [Listener at localhost/46495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 13:55:41,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 13:55:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:41,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 13:55:41,287 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307741287"}]},"ts":"1681307741287"} 2023-04-12 13:55:41,288 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 13:55:41,382 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 13:55:41,383 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, UNASSIGN}] 2023-04-12 13:55:41,385 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, UNASSIGN 2023-04-12 13:55:41,385 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:41,385 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307741385"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307741385"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307741385"}]},"ts":"1681307741385"} 2023-04-12 13:55:41,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 13:55:41,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:55:41,537 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:41,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7eb722b369ab23b3fdff931ce8c3cac2, disabling compactions & flushes 2023-04-12 13:55:41,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. after waiting 0 ms 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:41,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7eb722b369ab23b3fdff931ce8c3cac2 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=A 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=B 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 7eb722b369ab23b3fdff931ce8c3cac2, store=C 2023-04-12 13:55:41,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:41,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b6e5269d9c6944feb38cc411bbcf27c7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b6e5269d9c6944feb38cc411bbcf27c7 2023-04-12 13:55:41,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=399, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/7abd49d3b862460888402486ac22906a 2023-04-12 13:55:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 13:55:41,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 13:55:41,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/19e5eed62f454036aff23e5f96247220 2023-04-12 13:55:42,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/465cb9d3484345c3a3889fb859e58b13 2023-04-12 13:55:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 13:55:42,390 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/A/7abd49d3b862460888402486ac22906a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/7abd49d3b862460888402486ac22906a 2023-04-12 13:55:42,394 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/7abd49d3b862460888402486ac22906a, entries=100, sequenceid=399, filesize=18.6 K 2023-04-12 13:55:42,394 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/B/19e5eed62f454036aff23e5f96247220 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/19e5eed62f454036aff23e5f96247220 2023-04-12 13:55:42,397 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/19e5eed62f454036aff23e5f96247220, entries=100, sequenceid=399, filesize=9.5 K 2023-04-12 13:55:42,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/.tmp/C/465cb9d3484345c3a3889fb859e58b13 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/465cb9d3484345c3a3889fb859e58b13 2023-04-12 13:55:42,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/465cb9d3484345c3a3889fb859e58b13, entries=100, sequenceid=399, filesize=9.5 K 2023-04-12 13:55:42,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 7eb722b369ab23b3fdff931ce8c3cac2 in 863ms, sequenceid=399, compaction requested=true 2023-04-12 13:55:42,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/recovered.edits/402.seqid, newMaxSeqId=402, maxSeqId=4 2023-04-12 13:55:42,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2. 2023-04-12 13:55:42,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7eb722b369ab23b3fdff931ce8c3cac2: 2023-04-12 13:55:42,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:42,411 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=7eb722b369ab23b3fdff931ce8c3cac2, regionState=CLOSED 2023-04-12 13:55:42,411 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307742411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307742411"}]},"ts":"1681307742411"} 2023-04-12 13:55:42,413 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-12 13:55:42,414 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 7eb722b369ab23b3fdff931ce8c3cac2, server=jenkins-hbase4.apache.org,35663,1681307616979 in 1.0260 sec 2023-04-12 13:55:42,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-12 13:55:42,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=7eb722b369ab23b3fdff931ce8c3cac2, UNASSIGN in 1.0310 sec 2023-04-12 13:55:42,415 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307742415"}]},"ts":"1681307742415"} 2023-04-12 13:55:42,416 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 13:55:42,417 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 13:55:42,419 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1380 sec 2023-04-12 13:55:43,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-12 13:55:43,390 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-12 13:55:43,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 13:55:43,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,393 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-12 13:55:43,394 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,398 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:43,400 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/recovered.edits] 2023-04-12 13:55:43,402 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/7abd49d3b862460888402486ac22906a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/7abd49d3b862460888402486ac22906a 2023-04-12 13:55:43,403 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/ae000806cac54d19ab646dd39032699e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/ae000806cac54d19ab646dd39032699e 2023-04-12 13:55:43,404 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/da9ae552c4f3422a889432e0ffc6aba0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/da9ae552c4f3422a889432e0ffc6aba0 2023-04-12 13:55:43,405 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f863f137b82d4287ae90b871350a626e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/A/f863f137b82d4287ae90b871350a626e 2023-04-12 13:55:43,407 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/19e5eed62f454036aff23e5f96247220 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/19e5eed62f454036aff23e5f96247220 2023-04-12 13:55:43,408 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/335ee5f8a69a46628eab9581ce2c6377 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/335ee5f8a69a46628eab9581ce2c6377 2023-04-12 13:55:43,409 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b9dd100637448b995a40c0f5d11f62f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/4b9dd100637448b995a40c0f5d11f62f 2023-04-12 13:55:43,410 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ccf9113978194fb3b1e33a4b0f5c875c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/B/ccf9113978194fb3b1e33a4b0f5c875c 2023-04-12 13:55:43,411 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/465cb9d3484345c3a3889fb859e58b13 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/465cb9d3484345c3a3889fb859e58b13 2023-04-12 13:55:43,412 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/611ff981da54416cbb741ad7ce284114 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/611ff981da54416cbb741ad7ce284114 2023-04-12 13:55:43,413 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6febb1785acc442aada6845e10f06320 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/6febb1785acc442aada6845e10f06320 2023-04-12 13:55:43,414 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/9e3173139e0e475096944e6b0e3dc87d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/C/9e3173139e0e475096944e6b0e3dc87d 2023-04-12 13:55:43,415 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/recovered.edits/402.seqid to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2/recovered.edits/402.seqid 2023-04-12 13:55:43,416 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/7eb722b369ab23b3fdff931ce8c3cac2 2023-04-12 13:55:43,416 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:55:43,416 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 13:55:43,417 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-12 13:55:43,419 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412096bf61e584a42a18e9cd4cb57ae83cc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412096bf61e584a42a18e9cd4cb57ae83cc 2023-04-12 13:55:43,420 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121a2ef3e4501b4021833929e341a32239 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121a2ef3e4501b4021833929e341a32239 2023-04-12 13:55:43,421 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121b26febdc3534715a4d7c88270b2d464 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121b26febdc3534715a4d7c88270b2d464 2023-04-12 13:55:43,422 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121ef1f879292b45a48e1b07fb1b1b3052 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121ef1f879292b45a48e1b07fb1b1b3052 2023-04-12 13:55:43,423 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122e42a03892124c5dbc933f9e39a76beb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122e42a03892124c5dbc933f9e39a76beb 2023-04-12 13:55:43,423 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041233e49a9925b749d987fd2da4a4eec651 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041233e49a9925b749d987fd2da4a4eec651 2023-04-12 13:55:43,424 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123bcdb61a33674658b846770bb8e82fc7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123bcdb61a33674658b846770bb8e82fc7 2023-04-12 13:55:43,425 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123c7fdd7c57f54da4ba0effa54cdf5bc0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304123c7fdd7c57f54da4ba0effa54cdf5bc0 2023-04-12 13:55:43,426 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412520e76708d56457680d28152c0768dca to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412520e76708d56457680d28152c0768dca 2023-04-12 13:55:43,427 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125507d36172cb455dbe5ec5b7d57279b8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125507d36172cb455dbe5ec5b7d57279b8 2023-04-12 13:55:43,427 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041265aa166d2a7d44b791a85a38102f1d15 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041265aa166d2a7d44b791a85a38102f1d15 2023-04-12 13:55:43,428 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128e36ec6738304472b82b1e9417388c6b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128e36ec6738304472b82b1e9417388c6b 2023-04-12 13:55:43,429 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129724b38a76b849f4a27515414ea97600 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129724b38a76b849f4a27515414ea97600 2023-04-12 13:55:43,430 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a93de667bd5b40d7805d6ac87348f24f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a93de667bd5b40d7805d6ac87348f24f 2023-04-12 13:55:43,431 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b6e5269d9c6944feb38cc411bbcf27c7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b6e5269d9c6944feb38cc411bbcf27c7 2023-04-12 13:55:43,432 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b9c0d8c8b3584c0eaa25ea837b5ed0ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b9c0d8c8b3584c0eaa25ea837b5ed0ab 2023-04-12 13:55:43,433 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d3ccae2a795141a3a781142d172d1e7c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d3ccae2a795141a3a781142d172d1e7c 2023-04-12 13:55:43,434 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d79f65281d9245169fe793e24beea2fd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d79f65281d9245169fe793e24beea2fd 2023-04-12 13:55:43,435 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d816d33870b54db39367c53bfbf8ddf7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d816d33870b54db39367c53bfbf8ddf7 2023-04-12 13:55:43,436 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412deaae8414cbb4ae59ea44440d85b31de to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412deaae8414cbb4ae59ea44440d85b31de 2023-04-12 13:55:43,437 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f534435702f84a3895b2f44787d65793 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f534435702f84a3895b2f44787d65793 2023-04-12 13:55:43,437 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f64e66695af84b5192ce9bd5f456003d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f64e66695af84b5192ce9bd5f456003d 2023-04-12 13:55:43,438 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f6c2367e85ed487e88a5c6bb188c4b76 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412f6c2367e85ed487e88a5c6bb188c4b76 2023-04-12 13:55:43,439 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 13:55:43,440 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,442 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 13:55:43,443 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 13:55:43,444 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,444 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 13:55:43,444 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681307743444"}]},"ts":"9223372036854775807"} 2023-04-12 13:55:43,446 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 13:55:43,446 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7eb722b369ab23b3fdff931ce8c3cac2, NAME => 'TestAcidGuarantees,,1681307709891.7eb722b369ab23b3fdff931ce8c3cac2.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 13:55:43,446 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 13:55:43,446 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681307743446"}]},"ts":"9223372036854775807"} 2023-04-12 13:55:43,447 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 13:55:43,449 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,449 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-12 13:55:43,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-12 13:55:43,495 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-12 13:55:43,506 INFO [Listener at localhost/46495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229 (was 224) - Thread LEAK? -, OpenFileDescriptor=524 (was 531), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=385 (was 366) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=633 (was 288) - AvailableMemoryMB LEAK? - 2023-04-12 13:55:43,521 INFO [Listener at localhost/46495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229, OpenFileDescriptor=524, MaxFileDescriptor=60000, SystemLoadAverage=385, ProcessCount=171, AvailableMemoryMB=633 2023-04-12 13:55:43,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:55:43,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:55:43,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 13:55:43,526 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:55:43,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-12 13:55:43,526 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:55:43,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 13:55:43,527 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:43,528 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 empty. 2023-04-12 13:55:43,528 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:43,528 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:55:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 13:55:43,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 13:55:43,941 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:55:43,942 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a97913894f2065758f48c5b0eb67d392, NAME => 'TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:55:43,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:55:43,949 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a97913894f2065758f48c5b0eb67d392, disabling compactions & flushes 2023-04-12 13:55:43,949 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:43,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:43,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. after waiting 0 ms 2023-04-12 13:55:43,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:43,950 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:43,950 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:43,952 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:55:43,952 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307743952"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307743952"}]},"ts":"1681307743952"} 2023-04-12 13:55:43,953 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:55:43,954 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:55:43,954 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307743954"}]},"ts":"1681307743954"} 2023-04-12 13:55:43,955 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 13:55:43,959 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, ASSIGN}] 2023-04-12 13:55:43,960 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, ASSIGN 2023-04-12 13:55:43,961 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:55:44,112 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=a97913894f2065758f48c5b0eb67d392, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,112 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307744112"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307744112"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307744112"}]},"ts":"1681307744112"} 2023-04-12 13:55:44,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:55:44,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 13:55:44,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a97913894f2065758f48c5b0eb67d392, NAME => 'TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:55:44,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:55:44,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,269 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,271 DEBUG [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A 2023-04-12 13:55:44,271 DEBUG [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A 2023-04-12 13:55:44,271 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:44,271 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a97913894f2065758f48c5b0eb67d392 columnFamilyName A 2023-04-12 13:55:44,271 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.HStore(310): Store=a97913894f2065758f48c5b0eb67d392/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:44,272 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,273 DEBUG [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B 2023-04-12 13:55:44,273 DEBUG [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B 2023-04-12 13:55:44,273 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:44,273 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a97913894f2065758f48c5b0eb67d392 columnFamilyName B 2023-04-12 13:55:44,273 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.HStore(310): Store=a97913894f2065758f48c5b0eb67d392/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:44,274 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,275 DEBUG [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C 2023-04-12 13:55:44,275 DEBUG [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C 2023-04-12 13:55:44,275 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:55:44,275 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a97913894f2065758f48c5b0eb67d392 columnFamilyName C 2023-04-12 13:55:44,275 INFO [StoreOpener-a97913894f2065758f48c5b0eb67d392-1] regionserver.HStore(310): Store=a97913894f2065758f48c5b0eb67d392/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:55:44,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:55:44,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:55:44,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a97913894f2065758f48c5b0eb67d392; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64008099, jitterRate=-0.04620499908924103}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:55:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:44,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., pid=49, masterSystemTime=1681307744265 2023-04-12 13:55:44,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,283 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=a97913894f2065758f48c5b0eb67d392, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,283 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307744283"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307744283"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307744283"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307744283"}]},"ts":"1681307744283"} 2023-04-12 13:55:44,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-12 13:55:44,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 in 171 msec 2023-04-12 13:55:44,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-12 13:55:44,286 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, ASSIGN in 326 msec 2023-04-12 13:55:44,287 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:55:44,287 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307744287"}]},"ts":"1681307744287"} 2023-04-12 13:55:44,288 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 13:55:44,290 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:55:44,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 767 msec 2023-04-12 13:55:44,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-12 13:55:44,631 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-12 13:55:44,631 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,636 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,637 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,639 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48370, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,640 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:55:44,641 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54772, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:55:44,643 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,649 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,649 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,653 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,653 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,657 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,657 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,661 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,661 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,668 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,668 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,672 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,672 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,675 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,676 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,683 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,683 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,688 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,688 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:55:44,695 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:55:44,702 DEBUG [hconnection-0x715ddaa7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:44,709 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48372, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,710 DEBUG [hconnection-0x7ffe7027-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,710 DEBUG [hconnection-0x5e709bb9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,710 DEBUG [hconnection-0x7148708b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,710 DEBUG [hconnection-0x16941ee5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,711 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,711 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48420, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,713 DEBUG [hconnection-0x4924d25e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,716 DEBUG [hconnection-0x57a029a3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,719 DEBUG [hconnection-0x5733984-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,719 DEBUG [hconnection-0x5c5cc029-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,722 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48410, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,720 DEBUG [hconnection-0x7a4a2a78-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:55:44,722 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,722 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,725 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48440, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,730 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,734 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:44,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:44,735 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:44,735 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:44,735 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:55:44,735 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:44,735 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:44,736 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,736 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,739 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,739 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:44,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:44,739 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,739 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:44,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:44,741 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:44,741 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:44,742 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:44,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:44,742 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:55:44,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:44,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:44,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:44,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:44,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:44,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:44,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:44,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:44,746 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,746 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:44,747 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:44,747 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:44,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:44,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:44,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:44,748 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:44,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:44,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcc149e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:44,753 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:44,753 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:44,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:44,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:44,758 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:44,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:44,760 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:44,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:44,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:44,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:44,761 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:44,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307804759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:44,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:44,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307744757, max timestamp=1681307744761Num uniques 859; 2023-04-12 13:55:44,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307744757, max timestamp=1681307744761Num uniques 859; 2023-04-12 13:55:44,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307744757, max timestamp=1681307744761Num uniques 859; 2023-04-12 13:55:44,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:44,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:44,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:44,765 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,767 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:44,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:44,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,767 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:44,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,768 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:44,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:44,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:44,777 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/4a9884008e4048eaa5e20f038aa4aa04 2023-04-12 13:55:44,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307804779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307804779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307804780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307804780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,868 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:44,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:44,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307804865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:44,880 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:44,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:44,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:44,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:44,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,883 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:44,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:44,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:44,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:44,884 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,884 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:44,885 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:44,885 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,886 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:44,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f274832[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:44,889 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:44,889 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307804886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307804886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:44,891 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:44,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:44,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:44,893 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:44,893 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:44,895 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:44,895 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:44,895 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:44,895 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:44,895 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:44,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307804890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307804890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,899 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,899 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,899 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,900 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:44,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:44,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:44,903 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:44,903 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:44,903 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:44,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:44,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:44,903 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,004 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:45,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:45,009 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:45,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:45,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:45,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,011 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,011 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:45,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:45,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:45,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:45,017 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,017 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:45,017 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:45,017 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,019 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd150b1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:45,021 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:45,021 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,023 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:45,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:45,023 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:45,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:45,029 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:45,029 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:45,029 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:45,029 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:45,029 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:45,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,031 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,035 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:45,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,043 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,043 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:45,043 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,043 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307805071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307805091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307805092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307805096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307805096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,143 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:45,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:45,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:45,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:45,147 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:45,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:45,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:45,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,149 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:45,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:45,151 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:45,151 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:45,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:45,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e02bff8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:45,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:45,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,157 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,157 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:45,157 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:45,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:45,157 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:45,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:45,168 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:45,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:45,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:45,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:45,168 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:45,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,170 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:45,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,176 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:45,176 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:45,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:45,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/15e6517df18647b281d72632a254dc78 2023-04-12 13:55:45,276 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:45,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:45,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:45,280 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:45,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:45,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:45,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,282 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:45,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:45,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:45,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:45,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:45,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:45,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:45,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:45,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:45,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:45,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1453642[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:45,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:45,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,291 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:45,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:45,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,294 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:45,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:45,299 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:45,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:45,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:45,299 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:45,299 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:45,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,302 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:45,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:45,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,308 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:45,308 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:45,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307805377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307805397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307805401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307805403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307805403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,408 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:45,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:45,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:45,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:45,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:45,412 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:45,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:45,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:45,413 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,413 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,414 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:45,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:45,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:45,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:45,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:45,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:45,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:45,416 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,416 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:45,416 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:45,416 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:45,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:45,418 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:45,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c68880[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:45,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:45,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,421 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:45,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:45,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:45,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:45,426 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:45,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:45,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:45,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:45,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:45,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:45,429 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:45,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:45,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:45,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,440 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:45,440 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,541 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:45,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:45,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:45,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:45,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:45,544 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:45,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:45,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:45,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,544 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,547 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:45,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:45,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:45,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:45,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:45,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:45,551 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53251871[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:45,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:45,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,553 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:45,553 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:45,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:45,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:45,556 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:45,558 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:45,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:45,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:45,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:45,558 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:45,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:45,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:45,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:45,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,567 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:45,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,567 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:45,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:45,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:45,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/072d39290f0b459f819e9f150a66c55a 2023-04-12 13:55:45,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/4a9884008e4048eaa5e20f038aa4aa04 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/4a9884008e4048eaa5e20f038aa4aa04 2023-04-12 13:55:45,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/4a9884008e4048eaa5e20f038aa4aa04, entries=100, sequenceid=13, filesize=9.3 K 2023-04-12 13:55:45,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/15e6517df18647b281d72632a254dc78 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/15e6517df18647b281d72632a254dc78 2023-04-12 13:55:45,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/15e6517df18647b281d72632a254dc78, entries=100, sequenceid=13, filesize=9.3 K 2023-04-12 13:55:45,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/072d39290f0b459f819e9f150a66c55a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/072d39290f0b459f819e9f150a66c55a 2023-04-12 13:55:45,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/072d39290f0b459f819e9f150a66c55a, entries=100, sequenceid=13, filesize=9.3 K 2023-04-12 13:55:45,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for a97913894f2065758f48c5b0eb67d392 in 908ms, sequenceid=13, compaction requested=false 2023-04-12 13:55:45,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:45,668 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:45,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:45,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:45,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:45,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:45,678 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:45,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:45,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:45,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:45,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:45,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:45,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:45,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:45,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:45,686 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-12 13:55:45,686 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:45,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:45,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:45,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:45,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307805907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307805908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307805909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307805909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:45,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:45,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307805911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307806015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307806017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307806017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,109 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/cb484242150e4648bae9b02192987397 2023-04-12 13:55:46,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307806219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307806223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307806223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,524 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/97abe845bbb44814b6428ae6d0faa09f 2023-04-12 13:55:46,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307806523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307806528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307806529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,554 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/caf6e284f97c46b093161208cca694b1 2023-04-12 13:55:46,561 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/cb484242150e4648bae9b02192987397 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/cb484242150e4648bae9b02192987397 2023-04-12 13:55:46,569 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/cb484242150e4648bae9b02192987397, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:55:46,569 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/97abe845bbb44814b6428ae6d0faa09f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/97abe845bbb44814b6428ae6d0faa09f 2023-04-12 13:55:46,573 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/97abe845bbb44814b6428ae6d0faa09f, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:55:46,574 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/caf6e284f97c46b093161208cca694b1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/caf6e284f97c46b093161208cca694b1 2023-04-12 13:55:46,581 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/caf6e284f97c46b093161208cca694b1, entries=150, sequenceid=49, filesize=11.6 K 2023-04-12 13:55:46,582 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a97913894f2065758f48c5b0eb67d392 in 896ms, sequenceid=49, compaction requested=false 2023-04-12 13:55:46,582 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:46,582 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:46,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:46,585 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,585 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:46,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e39092[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:46,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:46,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:46,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:46,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:46,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:46,597 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:46,597 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:46,597 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,597 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:46,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:46,600 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:46,600 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:46,600 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:46,600 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:46,600 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:46,603 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,603 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:46,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,604 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:46,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:46,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:46,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,610 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,610 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:46,610 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:46,610 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:46,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:46,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:46,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:46,610 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:46,610 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:46,611 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,611 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:46,611 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,611 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:46,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:46,611 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:46,611 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,711 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:46,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:46,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:46,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:46,716 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:46,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:46,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:46,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:46,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,716 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:46,718 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:46,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:46,719 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:46,719 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:46,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:46,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:46,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:46,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:46,721 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:46,721 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:46,721 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:46,721 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:46,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:46,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:46,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307746964, max timestamp=1681307746974Num uniques 859; 2023-04-12 13:55:46,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307746964, max timestamp=1681307746974Num uniques 859; 2023-04-12 13:55:46,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:46,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307746964, max timestamp=1681307746974Num uniques 859; 2023-04-12 13:55:46,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:46,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:47,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307807028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307807029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307807032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307807037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307807039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,131 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/9efefa4c53e74978ab067a3f00891d7f 2023-04-12 13:55:47,141 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/84862e704ceb4f048301b748ddd717bc 2023-04-12 13:55:47,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307807137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307807137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,159 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/d48e814ec1af46ec88e4c55513b99fab 2023-04-12 13:55:47,163 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/9efefa4c53e74978ab067a3f00891d7f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/9efefa4c53e74978ab067a3f00891d7f 2023-04-12 13:55:47,166 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/9efefa4c53e74978ab067a3f00891d7f, entries=150, sequenceid=60, filesize=11.6 K 2023-04-12 13:55:47,166 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/84862e704ceb4f048301b748ddd717bc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/84862e704ceb4f048301b748ddd717bc 2023-04-12 13:55:47,173 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/84862e704ceb4f048301b748ddd717bc, entries=150, sequenceid=60, filesize=11.6 K 2023-04-12 13:55:47,174 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/d48e814ec1af46ec88e4c55513b99fab as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d48e814ec1af46ec88e4c55513b99fab 2023-04-12 13:55:47,185 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d48e814ec1af46ec88e4c55513b99fab, entries=150, sequenceid=60, filesize=11.6 K 2023-04-12 13:55:47,188 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a97913894f2065758f48c5b0eb67d392 in 465ms, sequenceid=60, compaction requested=true 2023-04-12 13:55:47,189 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:47,189 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:47,191 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:47,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:47,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403c4cea[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:47,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:47,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:47,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:47,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:47,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:47,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:47,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:47,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:47,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:47,199 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:47,199 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:47,199 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:47,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,202 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:47,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:47,203 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,205 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:47,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:47,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:47,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,207 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:47,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:47,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:47,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:47,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:47,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:47,208 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:47,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,208 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,208 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:47,308 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:47,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:47,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:47,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:47,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:47,311 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:47,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:47,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:47,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:47,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:47,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:47,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:47,318 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:47,319 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:47,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:47,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:47,319 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 13:55:47,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:47,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:47,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:47,319 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:47,320 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:47,320 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:47,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:47,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:47,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307807378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307807378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307807480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307807485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307807692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307807697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,737 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/41a60127b12146979f9dba84a9384694 2023-04-12 13:55:47,748 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/2bc2b328672649b4a53e0e427b634e58 2023-04-12 13:55:47,759 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/8aa8de00dbf04c52825f3a732a5ff7a3 2023-04-12 13:55:47,765 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/41a60127b12146979f9dba84a9384694 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/41a60127b12146979f9dba84a9384694 2023-04-12 13:55:47,768 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/41a60127b12146979f9dba84a9384694, entries=150, sequenceid=95, filesize=11.6 K 2023-04-12 13:55:47,773 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/2bc2b328672649b4a53e0e427b634e58 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2bc2b328672649b4a53e0e427b634e58 2023-04-12 13:55:47,776 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2bc2b328672649b4a53e0e427b634e58, entries=150, sequenceid=95, filesize=11.6 K 2023-04-12 13:55:47,777 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/8aa8de00dbf04c52825f3a732a5ff7a3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8aa8de00dbf04c52825f3a732a5ff7a3 2023-04-12 13:55:47,780 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8aa8de00dbf04c52825f3a732a5ff7a3, entries=150, sequenceid=95, filesize=11.6 K 2023-04-12 13:55:47,781 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a97913894f2065758f48c5b0eb67d392 in 462ms, sequenceid=95, compaction requested=true 2023-04-12 13:55:47,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:47,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:47,786 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106e4ee5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:47,788 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:47,788 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,789 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:47,789 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:47,789 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,789 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:47,790 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:47,790 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:47,793 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:47,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:47,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:47,793 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:47,793 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:47,794 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,794 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:47,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,795 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:47,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:47,796 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,797 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:47,800 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:47,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:47,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:47,800 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:47,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:47,801 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:47,801 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:47,901 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:47,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:47,906 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:47,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:47,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:47,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:47,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,906 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,908 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:47,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:47,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:47,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:47,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:47,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:47,909 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:47,910 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:47,938 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 13:55:48,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:48,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307748037, max timestamp=1681307748049Num uniques 859; 2023-04-12 13:55:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307748037, max timestamp=1681307748049Num uniques 859; 2023-04-12 13:55:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307748037, max timestamp=1681307748049Num uniques 859; 2023-04-12 13:55:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:48,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:48,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307808075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307808075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307808076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307808077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307808078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307808188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307808188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307808189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307808190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307808190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,319 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c472a1449525430ea96d8d8aedf64e46 2023-04-12 13:55:48,332 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/20537c07b4d8405681f39759dd09e08b 2023-04-12 13:55:48,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307808392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307808392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307808396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307808397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307808398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307808699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307808700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307808706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307808706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307808711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,747 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/4cf3eef06f99484dacf265abfc7b0f84 2023-04-12 13:55:48,751 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c472a1449525430ea96d8d8aedf64e46 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c472a1449525430ea96d8d8aedf64e46 2023-04-12 13:55:48,754 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c472a1449525430ea96d8d8aedf64e46, entries=150, sequenceid=106, filesize=11.6 K 2023-04-12 13:55:48,755 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/20537c07b4d8405681f39759dd09e08b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/20537c07b4d8405681f39759dd09e08b 2023-04-12 13:55:48,762 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/20537c07b4d8405681f39759dd09e08b, entries=150, sequenceid=106, filesize=11.6 K 2023-04-12 13:55:48,763 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/4cf3eef06f99484dacf265abfc7b0f84 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4cf3eef06f99484dacf265abfc7b0f84 2023-04-12 13:55:48,768 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4cf3eef06f99484dacf265abfc7b0f84, entries=150, sequenceid=106, filesize=11.6 K 2023-04-12 13:55:48,768 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a97913894f2065758f48c5b0eb67d392 in 858ms, sequenceid=106, compaction requested=true 2023-04-12 13:55:48,770 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:48,771 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,771 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:48,772 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,773 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,773 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf32a09[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:48,775 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:48,775 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,776 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:48,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:48,776 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:48,776 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,776 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:48,778 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:48,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:48,778 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:48,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:48,781 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:48,781 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:48,781 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:48,781 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:48,781 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:48,782 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:48,782 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,783 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:48,783 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:48,784 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:48,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:48,787 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:48,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:48,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,787 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:48,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:48,788 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:48,787 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:48,788 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:48,888 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:48,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:48,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:48,892 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:48,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:48,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:48,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:48,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:48,894 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:48,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:48,894 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:48,894 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:48,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:48,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:48,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:48,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:48,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:48,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:48,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:48,896 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-12 13:55:48,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:48,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:48,909 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/5086e2ac8176404f9a55ad19be524473 2023-04-12 13:55:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:49,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:49,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307809224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307809228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307809229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307809230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307809233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,319 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/93109353cc694cb387fbcf8003720104 2023-04-12 13:55:49,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307809334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307809343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307809343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307809343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307809347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307809540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307809552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307809552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307809553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307809554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,729 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/590091183bfa478d8dbdb19f844acd59 2023-04-12 13:55:49,733 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/5086e2ac8176404f9a55ad19be524473 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5086e2ac8176404f9a55ad19be524473 2023-04-12 13:55:49,736 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5086e2ac8176404f9a55ad19be524473, entries=150, sequenceid=142, filesize=11.8 K 2023-04-12 13:55:49,736 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/93109353cc694cb387fbcf8003720104 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/93109353cc694cb387fbcf8003720104 2023-04-12 13:55:49,739 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/93109353cc694cb387fbcf8003720104, entries=150, sequenceid=142, filesize=11.8 K 2023-04-12 13:55:49,740 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/590091183bfa478d8dbdb19f844acd59 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/590091183bfa478d8dbdb19f844acd59 2023-04-12 13:55:49,743 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/590091183bfa478d8dbdb19f844acd59, entries=150, sequenceid=142, filesize=11.8 K 2023-04-12 13:55:49,743 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a97913894f2065758f48c5b0eb67d392 in 847ms, sequenceid=142, compaction requested=true 2023-04-12 13:55:49,743 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:49,743 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:49,743 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:49,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:49,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:49,744 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:49,745 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,745 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:49,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:49,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe050e5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:49,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:49,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,748 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:49,750 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,750 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:49,751 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:49,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:49,752 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:49,752 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:49,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:49,754 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:49,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:49,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:49,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:49,754 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:49,757 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:49,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:49,762 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,764 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:49,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:49,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:49,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:49,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:49,767 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:49,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,767 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:49,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,767 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:49,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:49,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:55:49,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:49,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:49,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:49,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:49,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:49,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:49,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/ea70dbbf8e9c419c9d9af89fc6741565 2023-04-12 13:55:49,868 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:49,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:49,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:49,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:49,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:49,871 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:49,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:49,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:49,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:49,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,873 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:49,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:49,873 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/4106650dfda348b58e957faad2648112 2023-04-12 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:49,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:49,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:49,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:49,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:49,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:49,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:49,877 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:49,883 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d1904ef[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:49,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:49,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,891 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:49,891 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:49,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/5b06b002ab084413a7ddcc2a03ddb165 2023-04-12 13:55:49,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:49,893 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:49,896 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:49,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:49,896 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:49,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:49,897 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:49,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:49,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:49,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/ea70dbbf8e9c419c9d9af89fc6741565 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ea70dbbf8e9c419c9d9af89fc6741565 2023-04-12 13:55:49,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,899 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,901 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:49,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ea70dbbf8e9c419c9d9af89fc6741565, entries=200, sequenceid=154, filesize=14.1 K 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:49,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:49,905 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:49,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:49,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:49,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:49,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,906 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:49,906 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:49,906 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:49,906 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:49,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/4106650dfda348b58e957faad2648112 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4106650dfda348b58e957faad2648112 2023-04-12 13:55:49,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4106650dfda348b58e957faad2648112, entries=150, sequenceid=154, filesize=11.8 K 2023-04-12 13:55:49,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/5b06b002ab084413a7ddcc2a03ddb165 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/5b06b002ab084413a7ddcc2a03ddb165 2023-04-12 13:55:49,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:49,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307809896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:49,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/5b06b002ab084413a7ddcc2a03ddb165, entries=150, sequenceid=154, filesize=11.8 K 2023-04-12 13:55:49,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a97913894f2065758f48c5b0eb67d392 in 70ms, sequenceid=154, compaction requested=true 2023-04-12 13:55:49,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:49,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:49,925 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:49,925 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:49,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:49,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:49,927 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83548 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:49,927 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81158 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:49,927 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:55:49,927 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:55:49,927 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:49,927 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:49,927 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/4a9884008e4048eaa5e20f038aa4aa04, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/cb484242150e4648bae9b02192987397, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/9efefa4c53e74978ab067a3f00891d7f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/41a60127b12146979f9dba84a9384694, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c472a1449525430ea96d8d8aedf64e46, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5086e2ac8176404f9a55ad19be524473, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ea70dbbf8e9c419c9d9af89fc6741565] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=81.6 K 2023-04-12 13:55:49,927 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/15e6517df18647b281d72632a254dc78, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/97abe845bbb44814b6428ae6d0faa09f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/84862e704ceb4f048301b748ddd717bc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2bc2b328672649b4a53e0e427b634e58, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/20537c07b4d8405681f39759dd09e08b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/93109353cc694cb387fbcf8003720104, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4106650dfda348b58e957faad2648112] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=79.3 K 2023-04-12 13:55:49,927 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4a9884008e4048eaa5e20f038aa4aa04, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681307744739 2023-04-12 13:55:49,927 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 15e6517df18647b281d72632a254dc78, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681307744739 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting cb484242150e4648bae9b02192987397, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307744776 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 97abe845bbb44814b6428ae6d0faa09f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307744776 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 9efefa4c53e74978ab067a3f00891d7f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681307745901 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 84862e704ceb4f048301b748ddd717bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681307745901 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 41a60127b12146979f9dba84a9384694, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681307747000 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2bc2b328672649b4a53e0e427b634e58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681307747000 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c472a1449525430ea96d8d8aedf64e46, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681307747361 2023-04-12 13:55:49,928 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 20537c07b4d8405681f39759dd09e08b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681307747361 2023-04-12 13:55:49,929 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5086e2ac8176404f9a55ad19be524473, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681307748073 2023-04-12 13:55:49,929 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 93109353cc694cb387fbcf8003720104, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681307748073 2023-04-12 13:55:49,929 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ea70dbbf8e9c419c9d9af89fc6741565, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681307749223 2023-04-12 13:55:49,929 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4106650dfda348b58e957faad2648112, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681307749223 2023-04-12 13:55:49,943 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#407 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:49,943 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#408 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307749907, max timestamp=1681307749945Num uniques 859; 2023-04-12 13:55:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307749907, max timestamp=1681307749945Num uniques 859; 2023-04-12 13:55:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307749907, max timestamp=1681307749945Num uniques 859; 2023-04-12 13:55:49,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:49,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:49,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-04-12 13:55:49,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:49,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:49,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:49,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:49,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:49,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:49,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:49,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/62bc6ec9ddde4ec59673600775d13cb3 2023-04-12 13:55:50,006 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,010 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307749977, max timestamp=1681307750010Num uniques 859; 2023-04-12 13:55:50,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:50,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307749977, max timestamp=1681307750010Num uniques 859; 2023-04-12 13:55:50,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:50,013 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307749977, max timestamp=1681307750010Num uniques 859; 2023-04-12 13:55:50,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,019 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,019 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,021 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe27ccf[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,024 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,025 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,029 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,029 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,030 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,030 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,030 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,032 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,038 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,041 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:50,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,041 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,041 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,042 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,041 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681307750048, max timestamp=1681307750105Num uniques 1068; 2023-04-12 13:55:50,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-12 13:55:50,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681307750048, max timestamp=1681307750105Num uniques 1068; 2023-04-12 13:55:50,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-12 13:55:50,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681307750048, max timestamp=1681307750105Num uniques 1068; 2023-04-12 13:55:50,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-12 13:55:50,142 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,147 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,147 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,155 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d33a658[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,157 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681307750143, max timestamp=1681307750163Num uniques 1177; 2023-04-12 13:55:50,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=17262, suffix heap size=150376,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=377 2023-04-12 13:55:50,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681307750143, max timestamp=1681307750163Num uniques 1177; 2023-04-12 13:55:50,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=17262, suffix heap size=150376,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=377 2023-04-12 13:55:50,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681307750143, max timestamp=1681307750163Num uniques 1177; 2023-04-12 13:55:50,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=17262, suffix heap size=150376,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=377 2023-04-12 13:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,166 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,167 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,167 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,167 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,167 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,168 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,173 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,177 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,177 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,177 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:50,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307810201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681307750196, max timestamp=1681307750218Num uniques 1236; 2023-04-12 13:55:50,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=15386, suffix heap size=156512,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=336 2023-04-12 13:55:50,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681307750196, max timestamp=1681307750218Num uniques 1236; 2023-04-12 13:55:50,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=15386, suffix heap size=156512,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=336 2023-04-12 13:55:50,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681307750196, max timestamp=1681307750218Num uniques 1236; 2023-04-12 13:55:50,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=15386, suffix heap size=156512,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=336 2023-04-12 13:55:50,277 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,280 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,282 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,283 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,283 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,285 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,285 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,285 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,285 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c9ac63[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681307750274, max timestamp=1681307750292Num uniques 1195; 2023-04-12 13:55:50,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-04-12 13:55:50,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681307750274, max timestamp=1681307750292Num uniques 1195; 2023-04-12 13:55:50,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-04-12 13:55:50,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=395, cellSize=18090, totalHeapSize=41320, min timestamp=1681307750274, max timestamp=1681307750292Num uniques 1195; 2023-04-12 13:55:50,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=18090, suffix heap size=152248,new segment heap size=41320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=395 2023-04-12 13:55:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,299 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,299 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,301 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,301 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,302 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,307 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,307 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:50,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,358 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/ef64cea80ba44102abc530ae2c8218d4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ef64cea80ba44102abc530ae2c8218d4 2023-04-12 13:55:50,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307810346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,362 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/bcf9a6840d7640b280b887702b66c968 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bcf9a6840d7640b280b887702b66c968 2023-04-12 13:55:50,363 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into ef64cea80ba44102abc530ae2c8218d4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:50,363 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:50,363 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=9, startTime=1681307749925; duration=0sec 2023-04-12 13:55:50,363 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:50,363 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:50,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48394 deadline: 1681307810347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307810348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,370 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81158 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:50,370 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:55:50,370 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,371 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/072d39290f0b459f819e9f150a66c55a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/caf6e284f97c46b093161208cca694b1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d48e814ec1af46ec88e4c55513b99fab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8aa8de00dbf04c52825f3a732a5ff7a3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4cf3eef06f99484dacf265abfc7b0f84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/590091183bfa478d8dbdb19f844acd59, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/5b06b002ab084413a7ddcc2a03ddb165] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=79.3 K 2023-04-12 13:55:50,371 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 072d39290f0b459f819e9f150a66c55a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681307744739 2023-04-12 13:55:50,371 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into bcf9a6840d7640b280b887702b66c968(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:50,371 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:50,372 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=9, startTime=1681307749925; duration=0sec 2023-04-12 13:55:50,372 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:50,372 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting caf6e284f97c46b093161208cca694b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681307744776 2023-04-12 13:55:50,372 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d48e814ec1af46ec88e4c55513b99fab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1681307745901 2023-04-12 13:55:50,375 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 8aa8de00dbf04c52825f3a732a5ff7a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681307747000 2023-04-12 13:55:50,376 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4cf3eef06f99484dacf265abfc7b0f84, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1681307747361 2023-04-12 13:55:50,376 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 590091183bfa478d8dbdb19f844acd59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1681307748073 2023-04-12 13:55:50,376 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5b06b002ab084413a7ddcc2a03ddb165, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681307749223 2023-04-12 13:55:50,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307810359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307810359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,387 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#411 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:50,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/5d164e070012416da933912cbf02806e 2023-04-12 13:55:50,407 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/294d5fed36e84feb9264bc91bd6af2ba 2023-04-12 13:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,411 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,412 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,415 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/62bc6ec9ddde4ec59673600775d13cb3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/62bc6ec9ddde4ec59673600775d13cb3 2023-04-12 13:55:50,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,417 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,417 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/62bc6ec9ddde4ec59673600775d13cb3, entries=200, sequenceid=175, filesize=14.1 K 2023-04-12 13:55:50,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/5d164e070012416da933912cbf02806e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/5d164e070012416da933912cbf02806e 2023-04-12 13:55:50,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4502e8e9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/5d164e070012416da933912cbf02806e, entries=150, sequenceid=175, filesize=11.8 K 2023-04-12 13:55:50,428 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,428 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,428 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/294d5fed36e84feb9264bc91bd6af2ba as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/294d5fed36e84feb9264bc91bd6af2ba 2023-04-12 13:55:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,431 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,431 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,434 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,434 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/294d5fed36e84feb9264bc91bd6af2ba, entries=150, sequenceid=175, filesize=11.8 K 2023-04-12 13:55:50,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=167.72 KB/171750 for a97913894f2065758f48c5b0eb67d392 in 495ms, sequenceid=175, compaction requested=false 2023-04-12 13:55:50,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,446 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,446 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,446 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307750347, max timestamp=1681307750467Num uniques 1254; 2023-04-12 13:55:50,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307750347, max timestamp=1681307750467Num uniques 1254; 2023-04-12 13:55:50,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=7050, suffix heap size=158384,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=154 2023-04-12 13:55:50,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=7050, suffix heap size=158384,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=154 2023-04-12 13:55:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:50,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=76.04 KB heapSize=205.50 KB 2023-04-12 13:55:50,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:50,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307750347, max timestamp=1681307750467Num uniques 1254; 2023-04-12 13:55:50,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=57430, new segment data size=7050, suffix heap size=158384,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1254, new segment cells count=154 2023-04-12 13:55:50,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:50,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307750516, max timestamp=1681307750522Num uniques 859; 2023-04-12 13:55:50,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307750516, max timestamp=1681307750522Num uniques 859; 2023-04-12 13:55:50,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:50,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307750516, max timestamp=1681307750522Num uniques 859; 2023-04-12 13:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:50,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:55:50,546 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,550 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,550 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,552 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,556 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9caf082[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,568 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,571 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,571 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,571 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307750546, max timestamp=1681307750570Num uniques 1018; 2023-04-12 13:55:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307750546, max timestamp=1681307750570Num uniques 1018; 2023-04-12 13:55:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307750546, max timestamp=1681307750570Num uniques 1018; 2023-04-12 13:55:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 13:55:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 13:55:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-12 13:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,577 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,577 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,579 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,580 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,582 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,588 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,588 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681307750638, max timestamp=1681307750649Num uniques 1077; 2023-04-12 13:55:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681307750638, max timestamp=1681307750649Num uniques 1077; 2023-04-12 13:55:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-12 13:55:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681307750638, max timestamp=1681307750649Num uniques 1077; 2023-04-12 13:55:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-12 13:55:50,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-12 13:55:50,688 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,691 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,695 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,696 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,697 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,697 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681307750684, max timestamp=1681307750698Num uniques 1136; 2023-04-12 13:55:50,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-12 13:55:50,701 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681307750684, max timestamp=1681307750698Num uniques 1136; 2023-04-12 13:55:50,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-12 13:55:50,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681307750684, max timestamp=1681307750698Num uniques 1136; 2023-04-12 13:55:50,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=13096, suffix heap size=146112,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=286 2023-04-12 13:55:50,702 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,703 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fb9685[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,706 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,706 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,707 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,710 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,710 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,712 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,713 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,723 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,723 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,723 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,723 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681307750720, max timestamp=1681307750758Num uniques 1145; 2023-04-12 13:55:50,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-12 13:55:50,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681307750720, max timestamp=1681307750758Num uniques 1145; 2023-04-12 13:55:50,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-12 13:55:50,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681307750720, max timestamp=1681307750758Num uniques 1145; 2023-04-12 13:55:50,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=13510, suffix heap size=147048,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=295 2023-04-12 13:55:50,796 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/2702a9f2332d4482ac88f3e3ccc94bee as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2702a9f2332d4482ac88f3e3ccc94bee 2023-04-12 13:55:50,800 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into 2702a9f2332d4482ac88f3e3ccc94bee(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:50,800 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:50,800 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=9, startTime=1681307749925; duration=0sec 2023-04-12 13:55:50,800 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:50,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48410 deadline: 1681307810788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48454 deadline: 1681307810789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48428 deadline: 1681307810790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:50,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48372 deadline: 1681307810790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681307750786, max timestamp=1681307750802Num uniques 1154; 2023-04-12 13:55:50,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-12 13:55:50,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681307750786, max timestamp=1681307750802Num uniques 1154; 2023-04-12 13:55:50,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-12 13:55:50,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681307750786, max timestamp=1681307750802Num uniques 1154; 2023-04-12 13:55:50,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-12 13:55:50,823 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,827 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,827 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,830 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,830 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,831 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,831 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,833 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442c8f6d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,834 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,835 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,835 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,839 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,839 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,841 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,841 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,842 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,848 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,849 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,850 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,850 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c5a1cc0d743e495db871de8f04d972af 2023-04-12 13:55:50,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/af1c7882dd3548ba874a073a47df805b 2023-04-12 13:55:50,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307750853, max timestamp=1681307750908Num uniques 1113; 2023-04-12 13:55:50,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-12 13:55:50,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307750853, max timestamp=1681307750908Num uniques 1113; 2023-04-12 13:55:50,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307750853, max timestamp=1681307750908Num uniques 1113; 2023-04-12 13:55:50,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-12 13:55:50,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-12 13:55:50,951 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:50,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:50,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681307750937, max timestamp=1681307750949Num uniques 1022; 2023-04-12 13:55:50,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-12 13:55:50,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7866, totalHeapSize=18128, min timestamp=1681307750937, max timestamp=1681307750949Num uniques 1022; 2023-04-12 13:55:50,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46796, new segment data size=7866, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-12 13:55:50,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681307750937, max timestamp=1681307750949Num uniques 1022; 2023-04-12 13:55:50,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-12 13:55:50,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:50,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:50,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:50,955 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:50,955 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,955 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:50,955 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:50,956 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,956 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,957 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,957 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,957 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,957 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:50,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:50,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:50,958 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:50,958 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:50,958 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:50,959 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,959 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:50,959 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:50,959 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:50,961 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,962 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,962 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,962 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16bb62a1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:50,963 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:50,963 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:50,965 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:50,966 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:50,968 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:50,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:50,970 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:50,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:50,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:50,970 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:50,970 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:50,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,972 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:50,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,974 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,975 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:50,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:50,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:50,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:50,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:50,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:50,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:50,979 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:50,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,979 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:50,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:50,979 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:50,979 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:50,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681307750965, max timestamp=1681307750986Num uniques 1031; 2023-04-12 13:55:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681307750965, max timestamp=1681307750986Num uniques 1031; 2023-04-12 13:55:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-12 13:55:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-12 13:55:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681307750965, max timestamp=1681307750986Num uniques 1031; 2023-04-12 13:55:50,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=12866, suffix heap size=135192,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=281 2023-04-12 13:55:51,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681307751008, max timestamp=1681307751034Num uniques 1140; 2023-04-12 13:55:51,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-12 13:55:51,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681307751008, max timestamp=1681307751034Num uniques 1140; 2023-04-12 13:55:51,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-12 13:55:51,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681307751008, max timestamp=1681307751034Num uniques 1140; 2023-04-12 13:55:51,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-12 13:55:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681307751048, max timestamp=1681307751072Num uniques 1049; 2023-04-12 13:55:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1681307751048, max timestamp=1681307751072Num uniques 1049; 2023-04-12 13:55:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-12 13:55:51,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-12 13:55:51,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681307751048, max timestamp=1681307751072Num uniques 1049; 2023-04-12 13:55:51,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-12 13:55:51,079 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:51,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:51,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:51,083 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,086 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,087 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,087 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:51,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:51,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:51,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:51,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,088 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:51,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:51,089 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:51,089 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,090 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:51,090 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:51,090 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:51,090 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:51,091 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,091 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,092 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:51,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48abde14[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:51,094 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:51,094 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,095 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:51,095 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:51,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,095 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:51,097 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,097 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:51,101 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:51,101 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:51,101 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:51,101 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:51,101 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:51,103 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,103 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,103 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,108 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,113 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,113 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:51,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:51,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681307751099, max timestamp=1681307751113Num uniques 1158; 2023-04-12 13:55:51,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-12 13:55:51,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:51,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681307751099, max timestamp=1681307751113Num uniques 1108; 2023-04-12 13:55:51,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-12 13:55:51,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681307751099, max timestamp=1681307751113Num uniques 1158; 2023-04-12 13:55:51,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=9520, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-12 13:55:51,118 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,119 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,119 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,119 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:51,119 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:51,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:51,119 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,119 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,119 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,119 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:51,119 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:51,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681307751137, max timestamp=1681307751149Num uniques 1067; 2023-04-12 13:55:51,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-12 13:55:51,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681307751137, max timestamp=1681307751149Num uniques 1067; 2023-04-12 13:55:51,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681307751137, max timestamp=1681307751149Num uniques 1067; 2023-04-12 13:55:51,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-12 13:55:51,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-12 13:55:51,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307751175, max timestamp=1681307751185Num uniques 1126; 2023-04-12 13:55:51,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-12 13:55:51,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307751175, max timestamp=1681307751185Num uniques 1126; 2023-04-12 13:55:51,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307751175, max timestamp=1681307751185Num uniques 1126; 2023-04-12 13:55:51,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-12 13:55:51,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-12 13:55:51,219 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:51,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681307751204, max timestamp=1681307751219Num uniques 1035; 2023-04-12 13:55:51,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-12 13:55:51,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681307751204, max timestamp=1681307751219Num uniques 1035; 2023-04-12 13:55:51,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-12 13:55:51,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1681307751204, max timestamp=1681307751219Num uniques 1035; 2023-04-12 13:55:51,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-04-12 13:55:51,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:51,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:51,224 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:51,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:51,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:51,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,227 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,227 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:51,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:51,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:51,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:51,228 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:51,228 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,229 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:51,229 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:51,229 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:51,229 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:51,230 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,231 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:51,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d01363a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:51,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:51,233 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,234 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:51,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:51,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:51,234 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,234 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:51,235 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,235 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:51,238 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:51,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:51,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:51,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:51,238 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:51,243 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,243 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,243 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,244 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,245 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:51,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:51,247 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,247 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,248 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:51,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,248 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:51,248 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:51,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:51,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681307751248, max timestamp=1681307751257Num uniques 1144; 2023-04-12 13:55:51,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681307751248, max timestamp=1681307751257Num uniques 1144; 2023-04-12 13:55:51,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-12 13:55:51,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681307751248, max timestamp=1681307751257Num uniques 1144; 2023-04-12 13:55:51,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-12 13:55:51,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=15754, suffix heap size=146944,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=344 2023-04-12 13:55:51,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307811293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307811294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/6d14c56b9b944f9987f736a678b5c71c 2023-04-12 13:55:51,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307811295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307811295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307811301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c5a1cc0d743e495db871de8f04d972af as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c5a1cc0d743e495db871de8f04d972af 2023-04-12 13:55:51,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c5a1cc0d743e495db871de8f04d972af, entries=200, sequenceid=284, filesize=14.3 K 2023-04-12 13:55:51,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/af1c7882dd3548ba874a073a47df805b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/af1c7882dd3548ba874a073a47df805b 2023-04-12 13:55:51,321 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/af1c7882dd3548ba874a073a47df805b, entries=150, sequenceid=284, filesize=11.9 K 2023-04-12 13:55:51,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/6d14c56b9b944f9987f736a678b5c71c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/6d14c56b9b944f9987f736a678b5c71c 2023-04-12 13:55:51,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/6d14c56b9b944f9987f736a678b5c71c, entries=150, sequenceid=284, filesize=11.9 K 2023-04-12 13:55:51,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.66 KB/68256, currentSize=161.02 KB/164880 for a97913894f2065758f48c5b0eb67d392 in 854ms, sequenceid=284, compaction requested=true 2023-04-12 13:55:51,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:51,326 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:51,326 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:51,326 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:51,326 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:51,326 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:51,329 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41328 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:51,329 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:55:51,329 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,329 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ef64cea80ba44102abc530ae2c8218d4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/62bc6ec9ddde4ec59673600775d13cb3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c5a1cc0d743e495db871de8f04d972af] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=40.4 K 2023-04-12 13:55:51,329 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:51,330 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:55:51,330 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ef64cea80ba44102abc530ae2c8218d4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681307749223 2023-04-12 13:55:51,330 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,330 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bcf9a6840d7640b280b887702b66c968, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/5d164e070012416da933912cbf02806e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/af1c7882dd3548ba874a073a47df805b] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=35.6 K 2023-04-12 13:55:51,330 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 62bc6ec9ddde4ec59673600775d13cb3, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1681307749907 2023-04-12 13:55:51,330 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bcf9a6840d7640b280b887702b66c968, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681307749223 2023-04-12 13:55:51,330 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c5a1cc0d743e495db871de8f04d972af, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681307750347 2023-04-12 13:55:51,330 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 5d164e070012416da933912cbf02806e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1681307749909 2023-04-12 13:55:51,331 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting af1c7882dd3548ba874a073a47df805b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681307750357 2023-04-12 13:55:51,341 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#416 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:51,341 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#417 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:51,348 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:51,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:51,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:51,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:51,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:51,353 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:51,353 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,353 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:51,353 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:51,354 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,354 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,355 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/4f8985ac1bc94733a5b716c61adce043 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4f8985ac1bc94733a5b716c61adce043 2023-04-12 13:55:51,355 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,355 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:51,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:51,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:51,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:51,357 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:51,357 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,357 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:51,357 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=161.02 KB heapSize=447.33 KB 2023-04-12 13:55:51,358 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:51,358 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:51,358 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:51,358 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:51,358 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:51,358 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:51,361 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into 4f8985ac1bc94733a5b716c61adce043(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:51,361 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:51,362 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=13, startTime=1681307751326; duration=0sec 2023-04-12 13:55:51,362 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:51,362 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:51,363 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36498 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:51,363 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:55:51,363 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,363 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2702a9f2332d4482ac88f3e3ccc94bee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/294d5fed36e84feb9264bc91bd6af2ba, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/6d14c56b9b944f9987f736a678b5c71c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=35.6 K 2023-04-12 13:55:51,363 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2702a9f2332d4482ac88f3e3ccc94bee, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1681307749223 2023-04-12 13:55:51,363 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 294d5fed36e84feb9264bc91bd6af2ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1681307749909 2023-04-12 13:55:51,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6d14c56b9b944f9987f736a678b5c71c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681307750357 2023-04-12 13:55:51,377 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:51,378 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c33594f41f56435188a10e4272e52c5b 2023-04-12 13:55:51,396 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/45fefa277daf4b2787d6d0ffc7b0a4e4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/45fefa277daf4b2787d6d0ffc7b0a4e4 2023-04-12 13:55:51,399 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into 45fefa277daf4b2787d6d0ffc7b0a4e4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:51,399 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:51,400 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=13, startTime=1681307751326; duration=0sec 2023-04-12 13:55:51,400 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:51,404 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/9ad8609c07cd4780831303c5a8d7d1b8 2023-04-12 13:55:51,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:51,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:51,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307811411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307811412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307811415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307811416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307811416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307811517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307811517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307811523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307811523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307811524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307811725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307811725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307811731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307811732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307811732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,758 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/51d200fe2e114742947f6827fab7bff6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/51d200fe2e114742947f6827fab7bff6 2023-04-12 13:55:51,762 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into 51d200fe2e114742947f6827fab7bff6(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:51,762 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:51,762 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=13, startTime=1681307751326; duration=0sec 2023-04-12 13:55:51,762 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:51,823 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/0e96d24e88f34e41b7129190abe0e1e8 2023-04-12 13:55:51,827 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c33594f41f56435188a10e4272e52c5b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c33594f41f56435188a10e4272e52c5b 2023-04-12 13:55:51,830 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c33594f41f56435188a10e4272e52c5b, entries=150, sequenceid=580, filesize=11.9 K 2023-04-12 13:55:51,831 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/9ad8609c07cd4780831303c5a8d7d1b8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/9ad8609c07cd4780831303c5a8d7d1b8 2023-04-12 13:55:51,834 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/9ad8609c07cd4780831303c5a8d7d1b8, entries=150, sequenceid=580, filesize=11.9 K 2023-04-12 13:55:51,835 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/0e96d24e88f34e41b7129190abe0e1e8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0e96d24e88f34e41b7129190abe0e1e8 2023-04-12 13:55:51,840 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0e96d24e88f34e41b7129190abe0e1e8, entries=150, sequenceid=580, filesize=11.9 K 2023-04-12 13:55:51,840 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~447.28 KB/458016, currentSize=33.54 KB/34350 for a97913894f2065758f48c5b0eb67d392 in 483ms, sequenceid=580, compaction requested=false 2023-04-12 13:55:51,841 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:51,841 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:51,842 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d768310[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:51,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:51,844 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,846 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:51,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:51,846 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:51,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:51,847 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:51,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:51,850 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:51,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:51,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:51,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:51,850 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:51,851 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,851 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,851 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:51,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:51,852 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:51,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,855 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:51,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:51,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:51,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:51,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:51,856 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:51,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:51,856 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:51,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:51,956 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:51,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:51,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:51,960 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:51,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:51,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:51,960 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,961 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,965 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:51,965 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:51,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:51,966 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:51,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:51,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:51,967 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:51,968 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:51,968 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-12 13:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:51,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:51,969 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:51,978 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/46e32bccd45143299259868996576d4a 2023-04-12 13:55:51,997 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/666fadebdfb6458d8056208560ba509b 2023-04-12 13:55:52,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:52,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:52,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307752051, max timestamp=1681307752057Num uniques 859; 2023-04-12 13:55:52,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:52,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307752051, max timestamp=1681307752057Num uniques 859; 2023-04-12 13:55:52,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:52,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307752051, max timestamp=1681307752057Num uniques 859; 2023-04-12 13:55:52,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:52,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307812075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307812078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307812079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307812080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307812082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307812182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307812187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307812189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307812189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307812190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307812390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307812390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307812393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307812393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307812398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,410 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/8b6dd3ce4f9d4cdebad8448800393a38 2023-04-12 13:55:52,418 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/46e32bccd45143299259868996576d4a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/46e32bccd45143299259868996576d4a 2023-04-12 13:55:52,422 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/46e32bccd45143299259868996576d4a, entries=100, sequenceid=591, filesize=9.5 K 2023-04-12 13:55:52,422 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/666fadebdfb6458d8056208560ba509b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/666fadebdfb6458d8056208560ba509b 2023-04-12 13:55:52,426 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/666fadebdfb6458d8056208560ba509b, entries=100, sequenceid=591, filesize=9.5 K 2023-04-12 13:55:52,430 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/8b6dd3ce4f9d4cdebad8448800393a38 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8b6dd3ce4f9d4cdebad8448800393a38 2023-04-12 13:55:52,448 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8b6dd3ce4f9d4cdebad8448800393a38, entries=100, sequenceid=591, filesize=9.5 K 2023-04-12 13:55:52,449 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for a97913894f2065758f48c5b0eb67d392 in 481ms, sequenceid=591, compaction requested=true 2023-04-12 13:55:52,449 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:52,449 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:52,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,451 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40954675[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:52,462 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:52,462 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,464 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:52,464 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:52,464 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,464 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:52,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,465 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:52,468 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:52,468 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:52,468 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:52,468 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:52,468 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:52,471 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,471 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:52,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,471 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:52,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:52,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:52,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:52,473 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:52,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:52,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:52,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:52,476 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:52,477 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:52,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:52,577 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:52,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:52,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:52,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:52,584 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:52,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:52,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:52,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:52,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:52,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:52,589 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:52,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:52,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:52,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:52,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:52,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:52,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:52,593 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:52,594 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:52,594 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:52,594 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:52,594 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-12 13:55:52,594 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:52,594 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:52,595 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:52,595 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:52,595 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:52,595 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:52,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:52,704 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:52,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307812705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307812706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307812706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307812707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307812710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307812815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307812818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307812818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307812818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307812818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,010 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c 2023-04-12 13:55:53,025 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/0d09670edc2b4022898b1be879a9b362 2023-04-12 13:55:53,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307813023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307813025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307813025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307813025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307813025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,036 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=628 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/2b4a63323598459b95b443514af84170 2023-04-12 13:55:53,040 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c 2023-04-12 13:55:53,045 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c, entries=150, sequenceid=628, filesize=11.9 K 2023-04-12 13:55:53,045 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/0d09670edc2b4022898b1be879a9b362 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/0d09670edc2b4022898b1be879a9b362 2023-04-12 13:55:53,050 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/0d09670edc2b4022898b1be879a9b362, entries=150, sequenceid=628, filesize=11.9 K 2023-04-12 13:55:53,051 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/2b4a63323598459b95b443514af84170 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2b4a63323598459b95b443514af84170 2023-04-12 13:55:53,054 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2b4a63323598459b95b443514af84170, entries=150, sequenceid=628, filesize=11.9 K 2023-04-12 13:55:53,055 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for a97913894f2065758f48c5b0eb67d392 in 461ms, sequenceid=628, compaction requested=true 2023-04-12 13:55:53,055 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:53,055 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:53,055 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:53,057 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:53,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:53,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56586b0d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:53,060 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,061 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:53,061 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,061 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:53,064 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:53,068 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:53,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:53,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:53,068 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:53,068 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:53,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,082 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:53,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,082 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,082 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:53,083 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:53,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:53,183 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:53,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:53,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:53,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:53,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:53,186 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:53,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:53,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:53,186 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:53,187 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,187 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:53,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:53,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:53,192 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,192 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:53,192 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-12 13:55:53,192 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:53,192 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:53,193 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:53,193 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:53,193 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:53,193 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:53,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:53,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307753353, max timestamp=1681307753362Num uniques 859; 2023-04-12 13:55:53,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:55:53,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307753353, max timestamp=1681307753362Num uniques 859; 2023-04-12 13:55:53,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:55:53,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307753353, max timestamp=1681307753362Num uniques 859; 2023-04-12 13:55:53,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:55:53,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307813384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307813388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307813389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307813390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307813395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307813496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307813496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307813501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307813501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307813501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,602 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/607620f0283b47ca8a199597282bb563 2023-04-12 13:55:53,619 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/cab1ffe5cc91492ab870a5fb4a09d51b 2023-04-12 13:55:53,638 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/3cf093efa3aa4936812ecef35e117038 2023-04-12 13:55:53,643 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/607620f0283b47ca8a199597282bb563 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/607620f0283b47ca8a199597282bb563 2023-04-12 13:55:53,649 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/607620f0283b47ca8a199597282bb563, entries=100, sequenceid=636, filesize=9.5 K 2023-04-12 13:55:53,649 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/cab1ffe5cc91492ab870a5fb4a09d51b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cab1ffe5cc91492ab870a5fb4a09d51b 2023-04-12 13:55:53,653 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cab1ffe5cc91492ab870a5fb4a09d51b, entries=100, sequenceid=636, filesize=9.5 K 2023-04-12 13:55:53,654 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/3cf093efa3aa4936812ecef35e117038 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3cf093efa3aa4936812ecef35e117038 2023-04-12 13:55:53,662 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3cf093efa3aa4936812ecef35e117038, entries=100, sequenceid=636, filesize=9.5 K 2023-04-12 13:55:53,664 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for a97913894f2065758f48c5b0eb67d392 in 472ms, sequenceid=636, compaction requested=true 2023-04-12 13:55:53,664 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:53,665 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:53,665 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:53,667 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,668 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:53,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:53,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cc271a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:53,669 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,670 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,671 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:53,671 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:53,671 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:53,671 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,671 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:53,673 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:53,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:53,676 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:53,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:53,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:53,677 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:53,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,678 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,679 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,686 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:53,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,686 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:53,686 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:53,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:53,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:53,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-12 13:55:53,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:53,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:53,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:53,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:53,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:53,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:55:53,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307813708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307813711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307813715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307813716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307813716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,786 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:53,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:53,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:53,790 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:53,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:53,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:53,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:53,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,790 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,791 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,791 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,792 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,792 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:53,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:53,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:53,793 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,793 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:53,794 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:53,794 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:53,796 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,796 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1b5450[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:53,797 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:53,797 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,799 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:53,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:53,799 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,799 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:53,806 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:53,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:53,809 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:53,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:53,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:53,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:53,809 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:53,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,813 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,815 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:53,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:53,815 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,815 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:53,815 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:53,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,816 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307813817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307813823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307813829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307813829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:53,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307813834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,916 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:53,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:53,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:53,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:53,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:53,919 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:53,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:53,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:53,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:53,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,920 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,921 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,921 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:53,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:53,923 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,923 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:53,923 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:53,923 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:53,923 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:53,929 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,929 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,930 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@215715a8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:53,931 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:53,931 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,933 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:53,933 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:53,933 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,933 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,934 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,934 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:53,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:53,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:53,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:53,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:53,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:53,937 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:53,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,943 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:53,944 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,946 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:53,948 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:53,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:53,948 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:53,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:53,949 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:53,949 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:53,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:53,949 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:53,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:54,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307814024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307814034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307814038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307814039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,049 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:54,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:54,053 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:54,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:54,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:54,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,053 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,054 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307814043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:54,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:54,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:54,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:54,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:54,057 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,059 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:54,059 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3442028c[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:54,061 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:54,061 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,064 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:54,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,064 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:54,064 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:54,066 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,066 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:54,070 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:54,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:54,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:54,070 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:54,070 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:54,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,072 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,073 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,074 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:54,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,077 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,077 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:54,077 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/33c0e5c12c1f4951a8965417266b6ea3 2023-04-12 13:55:54,177 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:54,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:54,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:54,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:54,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:54,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:54,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,181 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,183 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,183 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:54,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:54,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:54,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:54,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:54,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:54,185 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,187 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,187 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272517e9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:54,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:54,189 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,190 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:54,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:54,190 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:54,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,190 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:54,192 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:54,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,192 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:54,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:54,194 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:54,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:54,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:54,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:54,194 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:54,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,197 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,197 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,203 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:54,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:54,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,203 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:54,203 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:54,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,303 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:54,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:54,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:54,307 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:54,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:54,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:54,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,308 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,309 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,309 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:54,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:54,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:54,310 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:54,310 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,310 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:54,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:54,311 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:54,311 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:54,312 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,312 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,313 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:54,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360d7e89[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:54,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:54,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,316 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:54,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:54,316 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:54,316 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,317 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:54,321 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:54,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:54,325 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:54,325 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:54,325 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:54,325 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:54,325 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:54,326 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,327 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,328 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,330 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:54,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,331 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,331 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:54,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:54,331 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307814330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307814344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307814344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307814346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307814358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,431 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:54,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:54,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:54,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:54,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:54,434 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:54,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:54,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:54,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,435 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:54,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:54,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:54,439 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:54,439 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,439 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:54,439 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:54,439 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:54,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:54,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,441 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:54,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feec7b5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:54,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:54,443 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:54,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:54,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:54,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:54,448 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:54,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:54,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:54,448 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:54,448 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:54,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,452 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,453 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:54,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:54,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,456 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:54,456 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,455 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:54,456 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:54,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/36147211179c494bb3572082898426ca 2023-04-12 13:55:54,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/68b62e98eb8241d6bf6478da279310d4 2023-04-12 13:55:54,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/33c0e5c12c1f4951a8965417266b6ea3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/33c0e5c12c1f4951a8965417266b6ea3 2023-04-12 13:55:54,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/33c0e5c12c1f4951a8965417266b6ea3, entries=200, sequenceid=674, filesize=14.3 K 2023-04-12 13:55:54,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/36147211179c494bb3572082898426ca as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/36147211179c494bb3572082898426ca 2023-04-12 13:55:54,556 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:54,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:54,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/36147211179c494bb3572082898426ca, entries=150, sequenceid=674, filesize=11.9 K 2023-04-12 13:55:54,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:54,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/68b62e98eb8241d6bf6478da279310d4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/68b62e98eb8241d6bf6478da279310d4 2023-04-12 13:55:54,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:54,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:54,559 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:54,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:54,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:54,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,561 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:54,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,562 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:54,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:54,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:54,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/68b62e98eb8241d6bf6478da279310d4, entries=150, sequenceid=674, filesize=11.9 K 2023-04-12 13:55:54,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for a97913894f2065758f48c5b0eb67d392 in 859ms, sequenceid=674, compaction requested=true 2023-04-12 13:55:54,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:54,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:54,567 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:55:54,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:54,567 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:55:54,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:54,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:54,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:54,568 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71010 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:55:54,568 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68570 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:55:54,568 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:55:54,568 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:55:54,568 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,568 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,568 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/51d200fe2e114742947f6827fab7bff6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c33594f41f56435188a10e4272e52c5b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/46e32bccd45143299259868996576d4a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/607620f0283b47ca8a199597282bb563, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/33c0e5c12c1f4951a8965417266b6ea3] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=69.3 K 2023-04-12 13:55:54,568 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4f8985ac1bc94733a5b716c61adce043, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/9ad8609c07cd4780831303c5a8d7d1b8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/666fadebdfb6458d8056208560ba509b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/0d09670edc2b4022898b1be879a9b362, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cab1ffe5cc91492ab870a5fb4a09d51b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/36147211179c494bb3572082898426ca] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=67.0 K 2023-04-12 13:55:54,569 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 51d200fe2e114742947f6827fab7bff6, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681307750357 2023-04-12 13:55:54,569 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4f8985ac1bc94733a5b716c61adce043, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681307750357 2023-04-12 13:55:54,569 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c33594f41f56435188a10e4272e52c5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681307751291 2023-04-12 13:55:54,569 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9ad8609c07cd4780831303c5a8d7d1b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681307751291 2023-04-12 13:55:54,569 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 46e32bccd45143299259868996576d4a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1681307751413 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 666fadebdfb6458d8056208560ba509b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1681307751413 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 888d7b8e6f2c46a1bcb6dd7ee629bd4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1681307752078 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 0d09670edc2b4022898b1be879a9b362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1681307752078 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 607620f0283b47ca8a199597282bb563, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1681307752705 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting cab1ffe5cc91492ab870a5fb4a09d51b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1681307752705 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 33c0e5c12c1f4951a8965417266b6ea3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1681307753383 2023-04-12 13:55:54,570 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 36147211179c494bb3572082898426ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1681307753387 2023-04-12 13:55:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:54,576 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,576 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:54,576 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 13:55:54,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:54,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:54,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:54,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:54,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:54,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:54,586 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#434 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:54,587 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#435 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:54,607 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/eaeda299b1da42a18c59cc66e3b3fd75 2023-04-12 13:55:54,611 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/ab7e1a8705ed4532b454673c0a236803 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab7e1a8705ed4532b454673c0a236803 2023-04-12 13:55:54,612 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/1cf1913ce2d54ddf92713ccf58975a2d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/1cf1913ce2d54ddf92713ccf58975a2d 2023-04-12 13:55:54,617 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into ab7e1a8705ed4532b454673c0a236803(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:54,617 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:54,617 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=10, startTime=1681307754567; duration=0sec 2023-04-12 13:55:54,617 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:54,617 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:55:54,619 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into 1cf1913ce2d54ddf92713ccf58975a2d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:54,619 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:54,619 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=10, startTime=1681307754567; duration=0sec 2023-04-12 13:55:54,619 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:54,619 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68570 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:55:54,619 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:55:54,619 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:54,619 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/45fefa277daf4b2787d6d0ffc7b0a4e4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0e96d24e88f34e41b7129190abe0e1e8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8b6dd3ce4f9d4cdebad8448800393a38, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2b4a63323598459b95b443514af84170, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3cf093efa3aa4936812ecef35e117038, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/68b62e98eb8241d6bf6478da279310d4] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=67.0 K 2023-04-12 13:55:54,620 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 45fefa277daf4b2787d6d0ffc7b0a4e4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681307750357 2023-04-12 13:55:54,620 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 0e96d24e88f34e41b7129190abe0e1e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681307751291 2023-04-12 13:55:54,620 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 8b6dd3ce4f9d4cdebad8448800393a38, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1681307751413 2023-04-12 13:55:54,622 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2b4a63323598459b95b443514af84170, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=628, earliestPutTs=1681307752078 2023-04-12 13:55:54,622 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3cf093efa3aa4936812ecef35e117038, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1681307752705 2023-04-12 13:55:54,622 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 68b62e98eb8241d6bf6478da279310d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1681307753387 2023-04-12 13:55:54,627 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/4aae9729427b45c29d3d1942ff7c403d 2023-04-12 13:55:54,634 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#438 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:54,860 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:54,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307754870, max timestamp=1681307754881Num uniques 859; 2023-04-12 13:55:54,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307754870, max timestamp=1681307754881Num uniques 859; 2023-04-12 13:55:54,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:55:54,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:55:54,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307754870, max timestamp=1681307754881Num uniques 859; 2023-04-12 13:55:54,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:55:54,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:54,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307814900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:54,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307754896, max timestamp=1681307754905Num uniques 1118; 2023-04-12 13:55:54,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 13:55:54,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307754896, max timestamp=1681307754905Num uniques 1118; 2023-04-12 13:55:54,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307754896, max timestamp=1681307754905Num uniques 1118; 2023-04-12 13:55:54,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 13:55:54,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 13:55:54,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681307754927, max timestamp=1681307754941Num uniques 1077; 2023-04-12 13:55:54,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-12 13:55:54,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681307754927, max timestamp=1681307754941Num uniques 1077; 2023-04-12 13:55:54,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-12 13:55:54,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681307754927, max timestamp=1681307754941Num uniques 1077; 2023-04-12 13:55:54,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=14972, suffix heap size=139976,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=327 2023-04-12 13:55:54,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681307754953, max timestamp=1681307754981Num uniques 1186; 2023-04-12 13:55:54,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681307754953, max timestamp=1681307754981Num uniques 1186; 2023-04-12 13:55:54,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-04-12 13:55:54,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-04-12 13:55:54,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681307754953, max timestamp=1681307754981Num uniques 1186; 2023-04-12 13:55:54,984 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=15386, suffix heap size=151312,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=336 2023-04-12 13:55:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681307755002, max timestamp=1681307755015Num uniques 1195; 2023-04-12 13:55:55,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681307755002, max timestamp=1681307755015Num uniques 1195; 2023-04-12 13:55:55,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-12 13:55:55,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681307755002, max timestamp=1681307755015Num uniques 1195; 2023-04-12 13:55:55,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-12 13:55:55,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=15800, suffix heap size=152248,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=345 2023-04-12 13:55:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307815011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,050 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=681 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/48317a2481e64c14bd5954245f337bb2 2023-04-12 13:55:55,056 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/eaeda299b1da42a18c59cc66e3b3fd75 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/eaeda299b1da42a18c59cc66e3b3fd75 2023-04-12 13:55:55,057 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/f3af831368844c7a9b171106ab302e84 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f3af831368844c7a9b171106ab302e84 2023-04-12 13:55:55,059 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/eaeda299b1da42a18c59cc66e3b3fd75, entries=100, sequenceid=681, filesize=9.5 K 2023-04-12 13:55:55,060 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/4aae9729427b45c29d3d1942ff7c403d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4aae9729427b45c29d3d1942ff7c403d 2023-04-12 13:55:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307815052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,066 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4aae9729427b45c29d3d1942ff7c403d, entries=100, sequenceid=681, filesize=9.5 K 2023-04-12 13:55:55,067 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/48317a2481e64c14bd5954245f337bb2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48317a2481e64c14bd5954245f337bb2 2023-04-12 13:55:55,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307815052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,068 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into f3af831368844c7a9b171106ab302e84(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:55,068 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:55,068 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=10, startTime=1681307754567; duration=0sec 2023-04-12 13:55:55,068 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:55,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307815054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307755042, max timestamp=1681307755052Num uniques 1204; 2023-04-12 13:55:55,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307755042, max timestamp=1681307755052Num uniques 1204; 2023-04-12 13:55:55,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-12 13:55:55,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-12 13:55:55,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307755042, max timestamp=1681307755052Num uniques 1204; 2023-04-12 13:55:55,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=7050, suffix heap size=153184,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=154 2023-04-12 13:55:55,077 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48317a2481e64c14bd5954245f337bb2, entries=100, sequenceid=681, filesize=9.5 K 2023-04-12 13:55:55,078 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=26.84 KB/27480 for a97913894f2065758f48c5b0eb67d392 in 502ms, sequenceid=681, compaction requested=false 2023-04-12 13:55:55,078 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:55,078 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:55,080 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1ce6a6[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:55,083 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:55,083 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:55,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,087 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,087 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:55,093 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:55,093 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:55,093 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:55,093 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:55,093 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:55,097 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,097 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,098 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,101 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=53.67 KB heapSize=146.39 KB 2023-04-12 13:55:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:55,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:55,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:55,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:55,104 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:55,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,107 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,107 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,107 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,109 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,109 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:55,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:55,110 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:55,110 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:55,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:55,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/17bbcfedea7741018bdffbfffea3db8d 2023-04-12 13:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307755152, max timestamp=1681307755175Num uniques 859; 2023-04-12 13:55:55,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307755152, max timestamp=1681307755175Num uniques 859; 2023-04-12 13:55:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307755152, max timestamp=1681307755175Num uniques 859; 2023-04-12 13:55:55,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:55,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307815196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307815196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,210 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:55,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307815197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307815198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307815199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:55,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:55,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:55,214 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:55,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:55,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:55,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,214 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,216 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,216 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,217 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:55,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:55,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:55,218 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:55,219 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,219 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:55,219 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:55,219 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:55,219 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:55,222 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4892273[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:55,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:55,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,225 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:55,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,225 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,227 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:55,227 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:55,232 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:55,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:55,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:55,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:55,232 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:55,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,238 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,238 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,239 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:55,241 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:55,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,242 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:55,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,242 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,242 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:55,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307815304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307815311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307815311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307815311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307815312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,342 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:55,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:55,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:55,345 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:55,345 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,345 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:55,345 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:55,346 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,346 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,347 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:55,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:55,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:55,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:55,349 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,349 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:55,349 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:55,349 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:55,356 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,356 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:55,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59aded5d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:55,357 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:55,357 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,360 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:55,360 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:55,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:55,361 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,361 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:55,361 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,361 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:55,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:55,364 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:55,364 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:55,364 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:55,364 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:55,364 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:55,365 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,365 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,365 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,366 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,368 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:55,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:55,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,372 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,372 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,372 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,371 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,371 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,372 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,372 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:55,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:55,372 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:55,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,472 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:55,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:55,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:55,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:55,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:55,476 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:55,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:55,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:55,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,479 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:55,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:55,479 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,480 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:55,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:55,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:55,480 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:55,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:55,483 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,484 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:55,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a74ab2[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:55,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:55,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,486 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:55,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:55,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:55,489 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:55,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,489 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:55,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:55,492 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:55,492 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:55,492 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:55,492 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:55,492 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:55,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,493 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:55,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:55,494 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,495 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:55,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,499 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:55,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,499 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:55,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,499 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:55,500 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:55,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:55,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307815512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307815518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307815518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307815520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307815526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/a9700e755d1040abb6595d90cd409f35 2023-04-12 13:55:55,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/67403f25746e458fa6b248dc0d8b9f6c 2023-04-12 13:55:55,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/17bbcfedea7741018bdffbfffea3db8d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/17bbcfedea7741018bdffbfffea3db8d 2023-04-12 13:55:55,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/17bbcfedea7741018bdffbfffea3db8d, entries=200, sequenceid=795, filesize=14.3 K 2023-04-12 13:55:55,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/a9700e755d1040abb6595d90cd409f35 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a9700e755d1040abb6595d90cd409f35 2023-04-12 13:55:55,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a9700e755d1040abb6595d90cd409f35, entries=150, sequenceid=795, filesize=11.9 K 2023-04-12 13:55:55,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/67403f25746e458fa6b248dc0d8b9f6c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67403f25746e458fa6b248dc0d8b9f6c 2023-04-12 13:55:55,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67403f25746e458fa6b248dc0d8b9f6c, entries=150, sequenceid=795, filesize=11.9 K 2023-04-12 13:55:55,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~146.34 KB/149856, currentSize=134.18 KB/137400 for a97913894f2065758f48c5b0eb67d392 in 471ms, sequenceid=795, compaction requested=true 2023-04-12 13:55:55,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:55,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:55,576 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:55,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:55,576 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:55,576 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:55,577 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37100 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:55,577 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:55:55,577 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,577 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab7e1a8705ed4532b454673c0a236803, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/eaeda299b1da42a18c59cc66e3b3fd75, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/17bbcfedea7741018bdffbfffea3db8d] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=36.2 K 2023-04-12 13:55:55,577 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:55,577 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:55:55,577 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ab7e1a8705ed4532b454673c0a236803, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1681307753387 2023-04-12 13:55:55,577 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,577 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/1cf1913ce2d54ddf92713ccf58975a2d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4aae9729427b45c29d3d1942ff7c403d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a9700e755d1040abb6595d90cd409f35] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=33.8 K 2023-04-12 13:55:55,578 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting eaeda299b1da42a18c59cc66e3b3fd75, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681307753715 2023-04-12 13:55:55,578 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1cf1913ce2d54ddf92713ccf58975a2d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1681307753387 2023-04-12 13:55:55,578 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 17bbcfedea7741018bdffbfffea3db8d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681307755042 2023-04-12 13:55:55,578 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4aae9729427b45c29d3d1942ff7c403d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681307753715 2023-04-12 13:55:55,579 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a9700e755d1040abb6595d90cd409f35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681307755042 2023-04-12 13:55:55,588 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:55,589 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#444 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:55,600 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:55,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:55,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:55,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:55,605 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:55,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:55,605 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:55,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,607 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:55,607 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,608 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,608 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:55,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:55,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:55,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:55,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:55,624 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:55,628 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:55,628 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,628 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:55,628 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-12 13:55:55,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:55,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:55,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:55,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:55,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:55,629 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:55,637 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/fa695ea0b16746d5a62e5c003ddf9b78 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fa695ea0b16746d5a62e5c003ddf9b78 2023-04-12 13:55:55,643 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into fa695ea0b16746d5a62e5c003ddf9b78(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:55,643 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:55,643 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=13, startTime=1681307755576; duration=0sec 2023-04-12 13:55:55,643 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:55,643 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:55:55,645 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34660 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:55:55,645 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:55:55,645 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:55,646 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f3af831368844c7a9b171106ab302e84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48317a2481e64c14bd5954245f337bb2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67403f25746e458fa6b248dc0d8b9f6c] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=33.8 K 2023-04-12 13:55:55,647 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f3af831368844c7a9b171106ab302e84, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=674, earliestPutTs=1681307753387 2023-04-12 13:55:55,647 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 48317a2481e64c14bd5954245f337bb2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=681, earliestPutTs=1681307753715 2023-04-12 13:55:55,648 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 67403f25746e458fa6b248dc0d8b9f6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681307755042 2023-04-12 13:55:55,661 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:55,665 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/a30dc3321e914924b93d568e3f3233b6 2023-04-12 13:55:55,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:55,823 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307815847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307815848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307815854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307815854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307815854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307815956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307815956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307815961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307815962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:55,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:55,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307815962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,027 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/eec945e727ab4bad90b63af494bbf2aa as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eec945e727ab4bad90b63af494bbf2aa 2023-04-12 13:55:56,031 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into eec945e727ab4bad90b63af494bbf2aa(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:56,031 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:56,031 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=13, startTime=1681307755576; duration=0sec 2023-04-12 13:55:56,031 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:56,091 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/b9d942e63e3e4ccf81983770a273c0fe as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b9d942e63e3e4ccf81983770a273c0fe 2023-04-12 13:55:56,095 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into b9d942e63e3e4ccf81983770a273c0fe(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:56,095 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:56,095 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=13, startTime=1681307755576; duration=0sec 2023-04-12 13:55:56,095 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:56,098 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/42bb2cd1272145c4b1b94b485cbffaab 2023-04-12 13:55:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307816159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307816159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307816165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307816166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307816166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307816465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307816465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307816472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307816472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307816472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,508 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/0188a5f380d14d1c9b77badf3a5b76aa 2023-04-12 13:55:56,512 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/a30dc3321e914924b93d568e3f3233b6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a30dc3321e914924b93d568e3f3233b6 2023-04-12 13:55:56,517 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a30dc3321e914924b93d568e3f3233b6, entries=150, sequenceid=829, filesize=11.9 K 2023-04-12 13:55:56,518 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/42bb2cd1272145c4b1b94b485cbffaab as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/42bb2cd1272145c4b1b94b485cbffaab 2023-04-12 13:55:56,530 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/42bb2cd1272145c4b1b94b485cbffaab, entries=150, sequenceid=829, filesize=11.9 K 2023-04-12 13:55:56,535 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/0188a5f380d14d1c9b77badf3a5b76aa as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0188a5f380d14d1c9b77badf3a5b76aa 2023-04-12 13:55:56,540 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0188a5f380d14d1c9b77badf3a5b76aa, entries=150, sequenceid=829, filesize=11.9 K 2023-04-12 13:55:56,541 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=60.38 KB/61830 for a97913894f2065758f48c5b0eb67d392 in 913ms, sequenceid=829, compaction requested=false 2023-04-12 13:55:56,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:56,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:56,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:56,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:56,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:56,552 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c62b5d6[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:56,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:56,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:56,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:56,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:56,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:56,557 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:56,561 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:56,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:56,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:56,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:56,561 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:56,563 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:56,563 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,563 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:56,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:56,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:56,564 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,565 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:56,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:56,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:56,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:56,576 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,576 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:56,575 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:56,576 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:56,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:56,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:56,676 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:56,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:56,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:56,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:56,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:56,679 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:56,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:56,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:56,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:56,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:56,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,681 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:56,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:56,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:56,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:56,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:56,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:56,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:56,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:56,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:56,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:56,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:56,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,683 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:56,683 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:56,683 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:56,684 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-12 13:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:56,684 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:56,694 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/39c148d6d1624534a76622bc04c78af5 2023-04-12 13:55:56,709 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/f1f52ab2e3f64940bafc1acd99adc56a 2023-04-12 13:55:56,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:56,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:56,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307816992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307816992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307816993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:56,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:56,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307816994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307816995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307817099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307817100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307817100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307817100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307817101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,120 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/f631e4bc92294062a0eebb879124c889 2023-04-12 13:55:57,124 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/39c148d6d1624534a76622bc04c78af5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/39c148d6d1624534a76622bc04c78af5 2023-04-12 13:55:57,128 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/39c148d6d1624534a76622bc04c78af5, entries=150, sequenceid=844, filesize=11.9 K 2023-04-12 13:55:57,128 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-12 13:55:57,129 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/f1f52ab2e3f64940bafc1acd99adc56a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/f1f52ab2e3f64940bafc1acd99adc56a 2023-04-12 13:55:57,132 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/f1f52ab2e3f64940bafc1acd99adc56a, entries=150, sequenceid=844, filesize=11.9 K 2023-04-12 13:55:57,133 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/f631e4bc92294062a0eebb879124c889 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f631e4bc92294062a0eebb879124c889 2023-04-12 13:55:57,136 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f631e4bc92294062a0eebb879124c889, entries=150, sequenceid=844, filesize=11.9 K 2023-04-12 13:55:57,137 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a97913894f2065758f48c5b0eb67d392 in 453ms, sequenceid=844, compaction requested=true 2023-04-12 13:55:57,137 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:57,137 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:57,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:57,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:57,139 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,139 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:57,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21b58893[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:57,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:57,142 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,144 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:57,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:57,145 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,145 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:57,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,148 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:57,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:57,151 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:57,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:57,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:57,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:57,151 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:57,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:57,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,153 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:57,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:57,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:57,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:57,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:57,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:57,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:57,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:57,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:57,159 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:57,159 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:57,259 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:57,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:57,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:57,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:57,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:57,262 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:57,262 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:57,262 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:57,262 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:57,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:57,265 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:57,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:57,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:57,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:57,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:57,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:57,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:57,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:57,268 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:57,268 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:57,268 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:55:57,269 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:57,269 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:57,269 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:57,269 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:57,269 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:57,269 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:57,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:57,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:57,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307817317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307817322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307817323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307817324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307817325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307817425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307817431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307817431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307817431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307817436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307817631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307817639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307817639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307817639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:57,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307817643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,678 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=864 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/e9f341a309ef49e9895a8586064f77a6 2023-04-12 13:55:57,697 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=864 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/e82cb676a24f4ae2a6e1a5ce03189a19 2023-04-12 13:55:57,713 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=864 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/67c6a172b2414b3292dd83aac5f34d51 2023-04-12 13:55:57,717 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/e9f341a309ef49e9895a8586064f77a6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/e9f341a309ef49e9895a8586064f77a6 2023-04-12 13:55:57,724 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/e9f341a309ef49e9895a8586064f77a6, entries=150, sequenceid=864, filesize=11.9 K 2023-04-12 13:55:57,725 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/e82cb676a24f4ae2a6e1a5ce03189a19 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/e82cb676a24f4ae2a6e1a5ce03189a19 2023-04-12 13:55:57,745 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/e82cb676a24f4ae2a6e1a5ce03189a19, entries=150, sequenceid=864, filesize=11.9 K 2023-04-12 13:55:57,746 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/67c6a172b2414b3292dd83aac5f34d51 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67c6a172b2414b3292dd83aac5f34d51 2023-04-12 13:55:57,749 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67c6a172b2414b3292dd83aac5f34d51, entries=150, sequenceid=864, filesize=11.9 K 2023-04-12 13:55:57,750 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for a97913894f2065758f48c5b0eb67d392 in 482ms, sequenceid=864, compaction requested=true 2023-04-12 13:55:57,750 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:57,751 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:57,753 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:57,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c45efd[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:57,755 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:57,755 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:57,756 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:57,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:57,758 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:57,758 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:57,759 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:57,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:57,762 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:57,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:57,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:57,763 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:57,763 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:57,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:57,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:57,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:57,769 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,770 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,774 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:57,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:57,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:57,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:57,775 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:57,775 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,775 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:57,776 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:57,876 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:57,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:57,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:57,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:57,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:57,880 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:57,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:57,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:57,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:57,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,882 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:57,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:57,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:57,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:57,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:57,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:57,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:57,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:57,884 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:57,896 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:57,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:57,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:57,896 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-12 13:55:57,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:57,896 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:57,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:57,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:57,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:57,897 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:57,920 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/bbdf3e9fe8854b09bc353859b6113b4c 2023-04-12 13:55:57,942 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/a4096c2c71fd40d6b44e23a5252f21fe 2023-04-12 13:55:57,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:57,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:58,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307817989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307817990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307817991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307817991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307817992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307818104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307818104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307818106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307818106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307818106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307818312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307818312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307818313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307818314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307818314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,372 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=877 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/0089e1adf9f54a29b6bef2bd1303f6a7 2023-04-12 13:55:58,377 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/bbdf3e9fe8854b09bc353859b6113b4c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/bbdf3e9fe8854b09bc353859b6113b4c 2023-04-12 13:55:58,381 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/bbdf3e9fe8854b09bc353859b6113b4c, entries=150, sequenceid=877, filesize=11.9 K 2023-04-12 13:55:58,382 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/a4096c2c71fd40d6b44e23a5252f21fe as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a4096c2c71fd40d6b44e23a5252f21fe 2023-04-12 13:55:58,393 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a4096c2c71fd40d6b44e23a5252f21fe, entries=150, sequenceid=877, filesize=11.9 K 2023-04-12 13:55:58,394 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/0089e1adf9f54a29b6bef2bd1303f6a7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0089e1adf9f54a29b6bef2bd1303f6a7 2023-04-12 13:55:58,400 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0089e1adf9f54a29b6bef2bd1303f6a7, entries=150, sequenceid=877, filesize=11.9 K 2023-04-12 13:55:58,401 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for a97913894f2065758f48c5b0eb67d392 in 505ms, sequenceid=877, compaction requested=true 2023-04-12 13:55:58,401 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:58,401 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:58,403 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0c746d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:58,405 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:58,405 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,408 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:58,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:58,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:58,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:58,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,409 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,410 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:58,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:58,414 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:58,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:58,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:58,414 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:58,414 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:58,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,419 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,421 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:58,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:58,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,425 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:58,425 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:58,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:58,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,525 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:58,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:58,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:58,529 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:58,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:58,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:58,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,530 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,536 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:58,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:58,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:58,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:58,539 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:58,539 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:58,556 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b404ea23f173404b8910fee95b7a617b 2023-04-12 13:55:58,572 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/34fe39d5533f4786b266cf0780a4308d 2023-04-12 13:55:58,586 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/e63180c5580b418597b04b5afd22bf36 2023-04-12 13:55:58,592 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b404ea23f173404b8910fee95b7a617b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b404ea23f173404b8910fee95b7a617b 2023-04-12 13:55:58,598 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b404ea23f173404b8910fee95b7a617b, entries=150, sequenceid=897, filesize=11.9 K 2023-04-12 13:55:58,599 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/34fe39d5533f4786b266cf0780a4308d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/34fe39d5533f4786b266cf0780a4308d 2023-04-12 13:55:58,603 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/34fe39d5533f4786b266cf0780a4308d, entries=150, sequenceid=897, filesize=11.9 K 2023-04-12 13:55:58,604 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/e63180c5580b418597b04b5afd22bf36 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/e63180c5580b418597b04b5afd22bf36 2023-04-12 13:55:58,614 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/e63180c5580b418597b04b5afd22bf36, entries=150, sequenceid=897, filesize=11.9 K 2023-04-12 13:55:58,615 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for a97913894f2065758f48c5b0eb67d392 in 76ms, sequenceid=897, compaction requested=true 2023-04-12 13:55:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:58,615 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:58,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:58,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5439b81f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:58,622 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:58,622 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,623 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:58,626 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:58,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:58,627 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:58,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:58,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:58,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:58,627 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:58,629 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,629 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,629 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,632 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,633 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:58,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:55:58,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:58,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:58,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:58,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:58,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:58,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:58,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:58,637 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,637 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,637 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:58,638 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:58,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307758651, max timestamp=1681307758670Num uniques 859; 2023-04-12 13:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:58,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307758651, max timestamp=1681307758670Num uniques 859; 2023-04-12 13:55:58,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681307758651, max timestamp=1681307758670Num uniques 859; 2023-04-12 13:55:58,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:58,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-12 13:55:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307818692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307818699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307818700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307818700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307818703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,738 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:58,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:58,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:58,741 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:58,741 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,741 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:58,741 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:58,742 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,742 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,743 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,744 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,746 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,746 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:58,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:58,747 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,747 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:58,747 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:58,747 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:58,747 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:58,750 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:58,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c5431b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:58,752 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:58,752 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,753 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,754 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:58,754 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:58,754 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:58,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,754 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:58,755 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,755 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:58,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:58,758 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:58,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:58,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:58,758 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:58,758 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:58,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,760 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,761 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,764 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,767 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:58,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:58,768 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:58,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,768 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:58,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,768 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307818804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307818809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307818811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307818811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:58,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307818812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,868 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:58,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:58,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:58,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:58,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:58,872 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:58,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:58,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:58,873 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,873 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,874 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:58,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:58,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:58,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:58,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:58,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:58,878 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:58,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc3dd68[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:58,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:58,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,882 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,882 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:58,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:58,883 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:58,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,883 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:58,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:58,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,885 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:58,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:58,887 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:58,887 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:58,887 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:58,887 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:58,887 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:58,889 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,889 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,889 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:58,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:58,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,891 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:58,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:58,894 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:58,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:58,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:58,894 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:58,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:58,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:58,895 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:58,895 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:58,895 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:58,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:58,995 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:58,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:58,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:58,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:58,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:58,999 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:58,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:58,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:58,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:58,999 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,000 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,001 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,001 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,001 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,001 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:59,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:59,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22227381[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:59,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:59,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,014 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:59,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:59,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:59,018 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:59,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:59,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:59,018 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:59,018 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:59,020 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,020 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,021 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307819015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307819019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,027 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:59,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:59,028 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:59,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,028 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:59,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307819024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307819025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307819025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,043 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=907 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/6ef89575eedb48c5943c387399bb25e2 2023-04-12 13:55:59,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=907 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/c98970ab0105476987ef7b4649a3dacc 2023-04-12 13:55:59,128 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:59,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:59,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:59,133 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:59,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:59,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:59,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,135 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,136 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,136 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,136 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:59,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:59,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:59,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:59,137 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:59,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:59,138 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:59,138 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:59,140 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,140 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc2de9a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:59,142 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:59,142 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,144 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:59,145 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,145 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:59,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,147 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:59,150 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:59,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:59,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:59,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:59,150 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:59,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,156 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,158 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:59,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,159 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,159 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:59,159 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:59,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,259 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:59,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:59,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:59,263 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:59,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:59,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:59,264 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,264 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,265 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,266 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:59,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:59,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:59,267 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,267 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:59,267 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:59,267 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,267 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:59,271 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,271 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,271 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,271 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:59,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:59,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ec8f62[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:59,273 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:59,273 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,274 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:59,274 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,274 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:59,277 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:59,279 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:59,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:59,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:59,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:59,281 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:59,283 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,283 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,283 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,285 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,288 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,288 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:59,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,290 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:59,290 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:59,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307819327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307819328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307819333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307819335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307819336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,391 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:59,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:59,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:59,394 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:59,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:59,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:59,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,396 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,396 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:59,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:55:59,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:59,398 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,398 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:59,398 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:59,398 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:55:59,398 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,399 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:55:59,399 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:55:59,403 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:55:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:55:59,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a36338a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:55:59,404 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:55:59,405 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,406 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:55:59,406 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,406 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:55:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:55:59,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:55:59,411 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:55:59,415 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:55:59,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:55:59,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:55:59,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:55:59,415 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:55:59,418 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,418 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,418 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:55:59,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,419 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,422 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,422 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,423 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:55:59,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:55:59,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:55:59,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:55:59,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:55:59,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,423 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:55:59,423 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:55:59,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:55:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:55:59,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=907 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/ebf23db91b5f496e89b45d04a02eddac 2023-04-12 13:55:59,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/6ef89575eedb48c5943c387399bb25e2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/6ef89575eedb48c5943c387399bb25e2 2023-04-12 13:55:59,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/6ef89575eedb48c5943c387399bb25e2, entries=300, sequenceid=907, filesize=19.0 K 2023-04-12 13:55:59,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/c98970ab0105476987ef7b4649a3dacc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c98970ab0105476987ef7b4649a3dacc 2023-04-12 13:55:59,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c98970ab0105476987ef7b4649a3dacc, entries=150, sequenceid=907, filesize=11.9 K 2023-04-12 13:55:59,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/ebf23db91b5f496e89b45d04a02eddac as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/ebf23db91b5f496e89b45d04a02eddac 2023-04-12 13:55:59,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/ebf23db91b5f496e89b45d04a02eddac, entries=150, sequenceid=907, filesize=11.9 K 2023-04-12 13:55:59,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a97913894f2065758f48c5b0eb67d392 in 869ms, sequenceid=907, compaction requested=true 2023-04-12 13:55:59,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:59,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:59,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:59,504 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:59,504 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:59,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:59,506 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 93270 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:59,506 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:55:59,506 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,506 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fa695ea0b16746d5a62e5c003ddf9b78, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a30dc3321e914924b93d568e3f3233b6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/39c148d6d1624534a76622bc04c78af5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/e9f341a309ef49e9895a8586064f77a6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/bbdf3e9fe8854b09bc353859b6113b4c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b404ea23f173404b8910fee95b7a617b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/6ef89575eedb48c5943c387399bb25e2] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=91.1 K 2023-04-12 13:55:59,507 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting fa695ea0b16746d5a62e5c003ddf9b78, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681307755042 2023-04-12 13:55:59,507 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85950 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:59,507 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:55:59,507 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a30dc3321e914924b93d568e3f3233b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1681307755196 2023-04-12 13:55:59,507 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,507 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eec945e727ab4bad90b63af494bbf2aa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/42bb2cd1272145c4b1b94b485cbffaab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/f1f52ab2e3f64940bafc1acd99adc56a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/e82cb676a24f4ae2a6e1a5ce03189a19, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a4096c2c71fd40d6b44e23a5252f21fe, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/34fe39d5533f4786b266cf0780a4308d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c98970ab0105476987ef7b4649a3dacc] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=83.9 K 2023-04-12 13:55:59,507 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 39c148d6d1624534a76622bc04c78af5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681307755834 2023-04-12 13:55:59,508 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting eec945e727ab4bad90b63af494bbf2aa, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681307755042 2023-04-12 13:55:59,508 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e9f341a309ef49e9895a8586064f77a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=864, earliestPutTs=1681307756987 2023-04-12 13:55:59,508 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 42bb2cd1272145c4b1b94b485cbffaab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1681307755196 2023-04-12 13:55:59,508 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bbdf3e9fe8854b09bc353859b6113b4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1681307757311 2023-04-12 13:55:59,509 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f1f52ab2e3f64940bafc1acd99adc56a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681307755834 2023-04-12 13:55:59,509 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b404ea23f173404b8910fee95b7a617b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=897, earliestPutTs=1681307757972 2023-04-12 13:55:59,509 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e82cb676a24f4ae2a6e1a5ce03189a19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=864, earliestPutTs=1681307756987 2023-04-12 13:55:59,509 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6ef89575eedb48c5943c387399bb25e2, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1681307758626 2023-04-12 13:55:59,509 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a4096c2c71fd40d6b44e23a5252f21fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1681307757311 2023-04-12 13:55:59,510 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 34fe39d5533f4786b266cf0780a4308d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=897, earliestPutTs=1681307757972 2023-04-12 13:55:59,510 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c98970ab0105476987ef7b4649a3dacc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1681307758627 2023-04-12 13:55:59,521 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#464 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:59,523 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:55:59,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:55:59,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:55:59,527 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#465 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:59,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:55:59,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:55:59,533 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:55:59,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:55:59,533 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:55:59,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,534 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,535 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:55:59,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:55:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:55:59,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,535 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:55:59,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:55:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:55:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:55:59,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:55:59,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:55:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:55:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:55:59,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:55:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:55:59,542 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-12 13:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:55:59,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:59,543 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:55:59,543 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:55:59,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:55:59,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:55:59,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307819847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307819848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307819848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307819857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307819863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,939 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/78aba42b17a9466ea7823217e64565e1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/78aba42b17a9466ea7823217e64565e1 2023-04-12 13:55:59,943 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into 78aba42b17a9466ea7823217e64565e1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:59,943 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:59,943 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=9, startTime=1681307759504; duration=0sec 2023-04-12 13:55:59,943 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:55:59,943 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 13:55:59,947 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85950 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 13:55:59,947 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:55:59,947 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:55:59,947 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b9d942e63e3e4ccf81983770a273c0fe, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0188a5f380d14d1c9b77badf3a5b76aa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f631e4bc92294062a0eebb879124c889, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67c6a172b2414b3292dd83aac5f34d51, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0089e1adf9f54a29b6bef2bd1303f6a7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/e63180c5580b418597b04b5afd22bf36, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/ebf23db91b5f496e89b45d04a02eddac] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=83.9 K 2023-04-12 13:55:59,948 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b9d942e63e3e4ccf81983770a273c0fe, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681307755042 2023-04-12 13:55:59,948 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 0188a5f380d14d1c9b77badf3a5b76aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1681307755196 2023-04-12 13:55:59,949 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f631e4bc92294062a0eebb879124c889, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681307755834 2023-04-12 13:55:59,949 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/eaa7cc36915d4018bb4b8e6f5d3477d8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eaa7cc36915d4018bb4b8e6f5d3477d8 2023-04-12 13:55:59,949 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 67c6a172b2414b3292dd83aac5f34d51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=864, earliestPutTs=1681307756987 2023-04-12 13:55:59,950 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 0089e1adf9f54a29b6bef2bd1303f6a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=877, earliestPutTs=1681307757311 2023-04-12 13:55:59,951 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e63180c5580b418597b04b5afd22bf36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=897, earliestPutTs=1681307757972 2023-04-12 13:55:59,951 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ebf23db91b5f496e89b45d04a02eddac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1681307758627 2023-04-12 13:55:59,955 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into eaa7cc36915d4018bb4b8e6f5d3477d8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:55:59,955 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:55:59,955 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=9, startTime=1681307759504; duration=0sec 2023-04-12 13:55:59,955 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:55:59,957 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c2caf69eff134092beadd8d8e03c9233 2023-04-12 13:55:59,965 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#467 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:55:59,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307819964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307819967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:55:59,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:55:59,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307819970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307820174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307820175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307820177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,381 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/093e21f26a7047b681a5a41f1dd199d0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/093e21f26a7047b681a5a41f1dd199d0 2023-04-12 13:56:00,385 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/640f748fd6c546e3abe64cf9475e352c 2023-04-12 13:56:00,386 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into 093e21f26a7047b681a5a41f1dd199d0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:00,386 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:00,386 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=9, startTime=1681307759504; duration=0sec 2023-04-12 13:56:00,386 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:00,399 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/fc4968080509482c87c429f32ff3ed89 2023-04-12 13:56:00,405 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/c2caf69eff134092beadd8d8e03c9233 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c2caf69eff134092beadd8d8e03c9233 2023-04-12 13:56:00,411 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c2caf69eff134092beadd8d8e03c9233, entries=150, sequenceid=942, filesize=11.9 K 2023-04-12 13:56:00,411 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/640f748fd6c546e3abe64cf9475e352c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/640f748fd6c546e3abe64cf9475e352c 2023-04-12 13:56:00,415 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/640f748fd6c546e3abe64cf9475e352c, entries=150, sequenceid=942, filesize=11.9 K 2023-04-12 13:56:00,416 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/fc4968080509482c87c429f32ff3ed89 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/fc4968080509482c87c429f32ff3ed89 2023-04-12 13:56:00,423 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/fc4968080509482c87c429f32ff3ed89, entries=150, sequenceid=942, filesize=11.9 K 2023-04-12 13:56:00,423 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for a97913894f2065758f48c5b0eb67d392 in 881ms, sequenceid=942, compaction requested=false 2023-04-12 13:56:00,423 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:00,423 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:00,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:00,426 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b39e136[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:00,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:00,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:00,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:00,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:00,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:00,434 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:00,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:00,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:00,434 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:00,434 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:00,436 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:00,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,443 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:00,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,443 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:00,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:00,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:56:00,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:00,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:00,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:00,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:00,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:00,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:00,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/1c84f05104f844ffa3b9458963103339 2023-04-12 13:56:00,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307820529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307760524, max timestamp=1681307760536Num uniques 859; 2023-04-12 13:56:00,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:00,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307760524, max timestamp=1681307760536Num uniques 859; 2023-04-12 13:56:00,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:00,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307760524, max timestamp=1681307760536Num uniques 859; 2023-04-12 13:56:00,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:00,543 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:00,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:00,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:00,546 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:00,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:00,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:00,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,547 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:00,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:00,551 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:00,552 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,552 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:00,552 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:00,552 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:00,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:00,554 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a2d8c5[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:00,558 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:00,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,559 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:00,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:00,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,560 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:00,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:00,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:00,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:00,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:00,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:00,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:00,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:00,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,571 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:00,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,571 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:00,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,572 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:00,572 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:00,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307820591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307820599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307820640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,672 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:00,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:00,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:00,675 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:00,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:00,675 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:00,676 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,676 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,677 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,677 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,678 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:00,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:00,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:00,679 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,679 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:00,680 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:00,680 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:00,680 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:00,682 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,682 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a19c4ec[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:00,686 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:00,686 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,687 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:00,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:00,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:00,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,687 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:00,688 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:00,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:00,691 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:00,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:00,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:00,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:00,691 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:00,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,696 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,698 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,701 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,703 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,704 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,704 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,704 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:00,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:00,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:00,704 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,704 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:00,704 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:00,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307820701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307820707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,804 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:00,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:00,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:00,808 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:00,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:00,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:00,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:00,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:00,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:00,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:00,811 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,812 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:00,812 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:00,812 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:00,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:00,814 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:00,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b81c46[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:00,816 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:00,816 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,819 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:00,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,819 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:00,820 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:00,825 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:00,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:00,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:00,825 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:00,825 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:00,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,828 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:00,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,830 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,831 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,837 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,837 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:00,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:00,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:00,837 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:00,837 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,838 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:00,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,838 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:00,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:00,838 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:00,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307820847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307820867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307820872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/d7acadd2ab664b62bbac5947e0556fa8 2023-04-12 13:56:00,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307820906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:00,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307820915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/48d4440ea9ee449ea25f8d40d1cbbd53 2023-04-12 13:56:00,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/1c84f05104f844ffa3b9458963103339 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/1c84f05104f844ffa3b9458963103339 2023-04-12 13:56:00,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/1c84f05104f844ffa3b9458963103339, entries=200, sequenceid=956, filesize=14.3 K 2023-04-12 13:56:00,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/d7acadd2ab664b62bbac5947e0556fa8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/d7acadd2ab664b62bbac5947e0556fa8 2023-04-12 13:56:00,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/d7acadd2ab664b62bbac5947e0556fa8, entries=150, sequenceid=956, filesize=11.9 K 2023-04-12 13:56:00,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/48d4440ea9ee449ea25f8d40d1cbbd53 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48d4440ea9ee449ea25f8d40d1cbbd53 2023-04-12 13:56:00,937 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48d4440ea9ee449ea25f8d40d1cbbd53, entries=150, sequenceid=956, filesize=11.9 K 2023-04-12 13:56:00,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a97913894f2065758f48c5b0eb67d392 in 455ms, sequenceid=956, compaction requested=true 2023-04-12 13:56:00,938 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:00,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:00,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:00,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:00,938 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:00,938 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:00,938 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:00,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:00,939 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:00,939 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:56:00,940 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,940 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eaa7cc36915d4018bb4b8e6f5d3477d8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/640f748fd6c546e3abe64cf9475e352c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/d7acadd2ab664b62bbac5947e0556fa8] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=36.6 K 2023-04-12 13:56:00,940 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:00,940 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:56:00,940 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,940 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/78aba42b17a9466ea7823217e64565e1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c2caf69eff134092beadd8d8e03c9233, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/1c84f05104f844ffa3b9458963103339] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=38.9 K 2023-04-12 13:56:00,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:00,940 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting eaa7cc36915d4018bb4b8e6f5d3477d8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1681307758627 2023-04-12 13:56:00,941 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 78aba42b17a9466ea7823217e64565e1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1681307758627 2023-04-12 13:56:00,941 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 640f748fd6c546e3abe64cf9475e352c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=942, earliestPutTs=1681307758688 2023-04-12 13:56:00,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:00,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:00,942 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:00,942 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c2caf69eff134092beadd8d8e03c9233, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=942, earliestPutTs=1681307758688 2023-04-12 13:56:00,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:00,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:00,942 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 1c84f05104f844ffa3b9458963103339, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681307759834 2023-04-12 13:56:00,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,942 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,943 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d7acadd2ab664b62bbac5947e0556fa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681307759834 2023-04-12 13:56:00,944 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,944 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:00,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:00,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:00,945 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:00,945 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:00,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:00,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:00,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:00,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:00,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:00,954 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:00,956 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#473 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:00,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:00,960 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:00,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:00,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:00,960 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-12 13:56:00,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:00,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:00,960 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:00,961 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:00,961 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:00,961 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:00,962 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#474 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:00,984 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=992 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b582e779db7d4a6f8df6da45921ea703 2023-04-12 13:56:00,997 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=992 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/2be7a79b9adc4bcb93da2e66bf0ecdb4 2023-04-12 13:56:01,010 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=992 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/f11b1e7a6c874d98a857df6df2cc6366 2023-04-12 13:56:01,013 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b582e779db7d4a6f8df6da45921ea703 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b582e779db7d4a6f8df6da45921ea703 2023-04-12 13:56:01,017 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b582e779db7d4a6f8df6da45921ea703, entries=150, sequenceid=992, filesize=11.9 K 2023-04-12 13:56:01,017 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/2be7a79b9adc4bcb93da2e66bf0ecdb4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2be7a79b9adc4bcb93da2e66bf0ecdb4 2023-04-12 13:56:01,020 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2be7a79b9adc4bcb93da2e66bf0ecdb4, entries=150, sequenceid=992, filesize=11.9 K 2023-04-12 13:56:01,021 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/f11b1e7a6c874d98a857df6df2cc6366 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f11b1e7a6c874d98a857df6df2cc6366 2023-04-12 13:56:01,024 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f11b1e7a6c874d98a857df6df2cc6366, entries=150, sequenceid=992, filesize=11.9 K 2023-04-12 13:56:01,025 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for a97913894f2065758f48c5b0eb67d392 in 65ms, sequenceid=992, compaction requested=true 2023-04-12 13:56:01,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:01,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:01,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:01,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:01,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:01,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:01,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,029 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,030 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:01,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:01,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454c6d16[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:01,031 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:01,031 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:01,035 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:01,035 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:01,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:01,040 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:01,040 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:01,040 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:01,040 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:01,040 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:01,042 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,042 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,042 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:01,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:01,043 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:01,045 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:01,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:01,048 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:01,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:01,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:01,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:01,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:01,049 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:01,049 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:01,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:01,149 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:01,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:01,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:01,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:01,152 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:01,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:01,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:01,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:01,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,153 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:01,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:01,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:01,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:01,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:01,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:01,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:01,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:01,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:01,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:01,156 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-12 13:56:01,157 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:01,157 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:01,157 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:01,157 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:01,157 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:01,157 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:01,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:01,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:01,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307761198, max timestamp=1681307761240Num uniques 859; 2023-04-12 13:56:01,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:01,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307761198, max timestamp=1681307761240Num uniques 859; 2023-04-12 13:56:01,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307761198, max timestamp=1681307761240Num uniques 859; 2023-04-12 13:56:01,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:01,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:01,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681307761270, max timestamp=1681307761286Num uniques 1068; 2023-04-12 13:56:01,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681307761270, max timestamp=1681307761286Num uniques 1068; 2023-04-12 13:56:01,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681307761270, max timestamp=1681307761286Num uniques 1068; 2023-04-12 13:56:01,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 13:56:01,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 13:56:01,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-12 13:56:01,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681307761321, max timestamp=1681307761344Num uniques 1127; 2023-04-12 13:56:01,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-12 13:56:01,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681307761321, max timestamp=1681307761344Num uniques 1127; 2023-04-12 13:56:01,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-12 13:56:01,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681307761321, max timestamp=1681307761344Num uniques 1127; 2023-04-12 13:56:01,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-12 13:56:01,374 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/5219dc0bb3a94855b4b2264f6f8fa7bc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5219dc0bb3a94855b4b2264f6f8fa7bc 2023-04-12 13:56:01,378 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into 5219dc0bb3a94855b4b2264f6f8fa7bc(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:01,378 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:01,378 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=13, startTime=1681307760938; duration=0sec 2023-04-12 13:56:01,378 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:01,378 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:56:01,379 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:56:01,379 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:56:01,380 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:01,380 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/093e21f26a7047b681a5a41f1dd199d0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/fc4968080509482c87c429f32ff3ed89, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48d4440ea9ee449ea25f8d40d1cbbd53, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f11b1e7a6c874d98a857df6df2cc6366] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=48.5 K 2023-04-12 13:56:01,380 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 093e21f26a7047b681a5a41f1dd199d0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1681307758627 2023-04-12 13:56:01,380 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting fc4968080509482c87c429f32ff3ed89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=942, earliestPutTs=1681307758688 2023-04-12 13:56:01,381 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 48d4440ea9ee449ea25f8d40d1cbbd53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681307759834 2023-04-12 13:56:01,381 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f11b1e7a6c874d98a857df6df2cc6366, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681307760580 2023-04-12 13:56:01,388 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/35ca0617d0c0430d8d2a595e62c70786 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35ca0617d0c0430d8d2a595e62c70786 2023-04-12 13:56:01,390 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:01,399 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into 35ca0617d0c0430d8d2a595e62c70786(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:01,399 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:01,399 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=13, startTime=1681307760938; duration=0sec 2023-04-12 13:56:01,400 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:01,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681307761379, max timestamp=1681307761400Num uniques 1086; 2023-04-12 13:56:01,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681307761379, max timestamp=1681307761400Num uniques 1086; 2023-04-12 13:56:01,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-12 13:56:01,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-12 13:56:01,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681307761379, max timestamp=1681307761400Num uniques 1086; 2023-04-12 13:56:01,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-12 13:56:01,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681307761441, max timestamp=1681307761462Num uniques 1045; 2023-04-12 13:56:01,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-12 13:56:01,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681307761441, max timestamp=1681307761462Num uniques 1045; 2023-04-12 13:56:01,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-12 13:56:01,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681307761441, max timestamp=1681307761462Num uniques 1045; 2023-04-12 13:56:01,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-12 13:56:01,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681307761496, max timestamp=1681307761506Num uniques 1104; 2023-04-12 13:56:01,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-12 13:56:01,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681307761496, max timestamp=1681307761506Num uniques 1104; 2023-04-12 13:56:01,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681307761496, max timestamp=1681307761506Num uniques 1104; 2023-04-12 13:56:01,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-12 13:56:01,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-12 13:56:01,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1681307761530, max timestamp=1681307761548Num uniques 1113; 2023-04-12 13:56:01,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1681307761530, max timestamp=1681307761548Num uniques 1063; 2023-04-12 13:56:01,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=16618, suffix heap size=138520,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=363 2023-04-12 13:56:01,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=16618, suffix heap size=143720,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=363 2023-04-12 13:56:01,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=363, cellSize=16618, totalHeapSize=37992, min timestamp=1681307761530, max timestamp=1681307761548Num uniques 1063; 2023-04-12 13:56:01,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=16618, suffix heap size=138520,new segment heap size=37992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=363 2023-04-12 13:56:01,570 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=997 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/d291abd9d19f4fdebc0f77f9daa38e8b 2023-04-12 13:56:01,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681307761580, max timestamp=1681307761591Num uniques 1222; 2023-04-12 13:56:01,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681307761580, max timestamp=1681307761591Num uniques 1222; 2023-04-12 13:56:01,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=12452, suffix heap size=155056,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=272 2023-04-12 13:56:01,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681307761580, max timestamp=1681307761591Num uniques 1222; 2023-04-12 13:56:01,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=12452, suffix heap size=155056,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=272 2023-04-12 13:56:01,594 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55962, new segment data size=12452, suffix heap size=155056,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1222, new segment cells count=272 2023-04-12 13:56:01,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681307761612, max timestamp=1681307761640Num uniques 1131; 2023-04-12 13:56:01,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-12 13:56:01,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681307761612, max timestamp=1681307761640Num uniques 1131; 2023-04-12 13:56:01,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-12 13:56:01,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681307761612, max timestamp=1681307761640Num uniques 1131; 2023-04-12 13:56:01,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-04-12 13:56:01,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681307761648, max timestamp=1681307761692Num uniques 1090; 2023-04-12 13:56:01,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-12 13:56:01,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681307761648, max timestamp=1681307761692Num uniques 1090; 2023-04-12 13:56:01,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-12 13:56:01,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681307761648, max timestamp=1681307761692Num uniques 1090; 2023-04-12 13:56:01,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-12 13:56:01,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681307761718, max timestamp=1681307761734Num uniques 1099; 2023-04-12 13:56:01,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-12 13:56:01,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681307761718, max timestamp=1681307761734Num uniques 1099; 2023-04-12 13:56:01,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-12 13:56:01,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681307761718, max timestamp=1681307761734Num uniques 1099; 2023-04-12 13:56:01,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-12 13:56:01,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1681307761762, max timestamp=1681307761779Num uniques 1108; 2023-04-12 13:56:01,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-04-12 13:56:01,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1681307761762, max timestamp=1681307761779Num uniques 1108; 2023-04-12 13:56:01,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=308, cellSize=14100, totalHeapSize=32272, min timestamp=1681307761762, max timestamp=1681307761779Num uniques 1058; 2023-04-12 13:56:01,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=14100, suffix heap size=143200,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=308 2023-04-12 13:56:01,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=14100, suffix heap size=138000,new segment heap size=32272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=308 2023-04-12 13:56:01,810 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/4f8f3e955fae4150a8959e1541a19d5f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4f8f3e955fae4150a8959e1541a19d5f 2023-04-12 13:56:01,817 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into 4f8f3e955fae4150a8959e1541a19d5f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:01,817 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:01,817 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=12, startTime=1681307760938; duration=0sec 2023-04-12 13:56:01,817 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681307761823, max timestamp=1681307761840Num uniques 1167; 2023-04-12 13:56:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-04-12 13:56:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681307761823, max timestamp=1681307761840Num uniques 1167; 2023-04-12 13:56:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681307761823, max timestamp=1681307761840Num uniques 1167; 2023-04-12 13:56:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-04-12 13:56:01,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53442, new segment data size=9932, suffix heap size=149336,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1167, new segment cells count=217 2023-04-12 13:56:01,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307761874, max timestamp=1681307761895Num uniques 1076; 2023-04-12 13:56:01,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307761874, max timestamp=1681307761895Num uniques 1076; 2023-04-12 13:56:01,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307761874, max timestamp=1681307761895Num uniques 1076; 2023-04-12 13:56:01,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-12 13:56:01,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-12 13:56:01,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-12 13:56:01,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681307761918, max timestamp=1681307761948Num uniques 1035; 2023-04-12 13:56:01,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-12 13:56:01,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307761918, max timestamp=1681307761948Num uniques 1035; 2023-04-12 13:56:01,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=6870, suffix heap size=135608,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=150 2023-04-12 13:56:01,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307761918, max timestamp=1681307761948Num uniques 1035; 2023-04-12 13:56:01,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=6870, suffix heap size=135608,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=150 2023-04-12 13:56:01,980 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=997 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/b2a2f0a4ffe9486fbf9fe606f0b93172 2023-04-12 13:56:02,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681307761991, max timestamp=1681307762002Num uniques 1094; 2023-04-12 13:56:02,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-12 13:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681307761991, max timestamp=1681307762002Num uniques 1009; 2023-04-12 13:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681307761991, max timestamp=1681307762002Num uniques 1009; 2023-04-12 13:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11174, suffix heap size=132904,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=244 2023-04-12 13:56:02,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11174, suffix heap size=132904,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=244 2023-04-12 13:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681307762039, max timestamp=1681307762063Num uniques 1103; 2023-04-12 13:56:02,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-12 13:56:02,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681307762039, max timestamp=1681307762063Num uniques 1103; 2023-04-12 13:56:02,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-12 13:56:02,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681307762039, max timestamp=1681307762063Num uniques 1103; 2023-04-12 13:56:02,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-12 13:56:02,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681307762079, max timestamp=1681307762151Num uniques 1112; 2023-04-12 13:56:02,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=11992, suffix heap size=143616,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=262 2023-04-12 13:56:02,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681307762079, max timestamp=1681307762151Num uniques 1012; 2023-04-12 13:56:02,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-12 13:56:02,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681307762079, max timestamp=1681307762151Num uniques 1012; 2023-04-12 13:56:02,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-12 13:56:02,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681307762193, max timestamp=1681307762202Num uniques 1121; 2023-04-12 13:56:02,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681307762193, max timestamp=1681307762202Num uniques 1121; 2023-04-12 13:56:02,206 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681307762193, max timestamp=1681307762202Num uniques 1121; 2023-04-12 13:56:02,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-12 13:56:02,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-12 13:56:02,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=7826, suffix heap size=144552,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=171 2023-04-12 13:56:02,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681307762218, max timestamp=1681307762246Num uniques 1030; 2023-04-12 13:56:02,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-12 13:56:02,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681307762218, max timestamp=1681307762246Num uniques 1030; 2023-04-12 13:56:02,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-12 13:56:02,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1681307762218, max timestamp=1681307762246Num uniques 1030; 2023-04-12 13:56:02,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=12820, suffix heap size=135088,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=280 2023-04-12 13:56:02,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681307762279, max timestamp=1681307762294Num uniques 1139; 2023-04-12 13:56:02,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681307762279, max timestamp=1681307762294Num uniques 1139; 2023-04-12 13:56:02,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-12 13:56:02,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-12 13:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1681307762279, max timestamp=1681307762294Num uniques 1139; 2023-04-12 13:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-12 13:56:02,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681307762328, max timestamp=1681307762343Num uniques 1098; 2023-04-12 13:56:02,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-12 13:56:02,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681307762328, max timestamp=1681307762343Num uniques 1098; 2023-04-12 13:56:02,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-12 13:56:02,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681307762328, max timestamp=1681307762343Num uniques 1098; 2023-04-12 13:56:02,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-12 13:56:02,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681307762370, max timestamp=1681307762390Num uniques 1157; 2023-04-12 13:56:02,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=11765, suffix heap size=148296,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=257 2023-04-12 13:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681307762370, max timestamp=1681307762390Num uniques 1157; 2023-04-12 13:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681307762370, max timestamp=1681307762390Num uniques 1157; 2023-04-12 13:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=11765, suffix heap size=148296,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=257 2023-04-12 13:56:02,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=11765, suffix heap size=148296,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=257 2023-04-12 13:56:02,397 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=997 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/3270fd3e08dc488ca47b06c07b618344 2023-04-12 13:56:02,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/d291abd9d19f4fdebc0f77f9daa38e8b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d291abd9d19f4fdebc0f77f9daa38e8b 2023-04-12 13:56:02,408 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d291abd9d19f4fdebc0f77f9daa38e8b, entries=100, sequenceid=997, filesize=9.5 K 2023-04-12 13:56:02,410 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/b2a2f0a4ffe9486fbf9fe606f0b93172 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b2a2f0a4ffe9486fbf9fe606f0b93172 2023-04-12 13:56:02,416 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b2a2f0a4ffe9486fbf9fe606f0b93172, entries=100, sequenceid=997, filesize=9.5 K 2023-04-12 13:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/3270fd3e08dc488ca47b06c07b618344 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3270fd3e08dc488ca47b06c07b618344 2023-04-12 13:56:02,434 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3270fd3e08dc488ca47b06c07b618344, entries=100, sequenceid=997, filesize=9.5 K 2023-04-12 13:56:02,435 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=127.47 KB/130530 for a97913894f2065758f48c5b0eb67d392 in 1279ms, sequenceid=997, compaction requested=true 2023-04-12 13:56:02,435 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:02,435 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:02,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:02,436 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:02,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=134.18 KB heapSize=375.80 KB 2023-04-12 13:56:02,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:02,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:02,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:02,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:02,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:02,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:02,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,438 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,440 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,441 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7598f18a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:02,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:02,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:02,445 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,445 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:02,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:02,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,446 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,446 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:02,449 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:02,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:02,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:02,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:02,449 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:02,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,451 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,452 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,453 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,463 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:02,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:02,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,463 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,463 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,463 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:02,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307822458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307822466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1413 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/a564465110f44b2e855c64c7fc380765 2023-04-12 13:56:02,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307822471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,563 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:02,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:02,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:02,567 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:02,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:02,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:02,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,568 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,569 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:02,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:02,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:02,572 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:02,574 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31e27571[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:02,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:02,576 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:02,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:02,580 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307822572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307822573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:02,584 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:02,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:02,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:02,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:02,584 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:02,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,586 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,587 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307822582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,592 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,592 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,593 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:02,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:02,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,593 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,593 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:02,594 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:02,704 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:02,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:02,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:02,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:02,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:02,708 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:02,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:02,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:02,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,710 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,711 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:02,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:02,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:02,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:02,719 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,719 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,720 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f91527[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:02,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:02,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,724 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:02,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:02,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:02,725 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:02,725 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,725 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,725 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,725 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:02,726 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:02,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:02,728 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:02,728 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:02,728 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:02,728 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:02,728 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:02,730 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,730 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,730 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,732 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,733 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,736 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:02,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,736 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:02,736 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:02,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307822782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307822783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307822793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,836 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:02,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:02,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:02,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:02,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:02,840 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:02,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:02,840 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:02,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:02,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:02,845 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,845 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:02,845 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:02,845 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:02,845 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:02,847 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,848 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fbf2f3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:02,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:02,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:02,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,851 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,853 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:02,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:02,856 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:02,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:02,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:02,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:02,856 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:02,858 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,858 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,858 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:02,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,858 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:02,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:02,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:02,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:02,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:02,865 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,864 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:02,865 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:02,865 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:02,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:02,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:02,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48428 deadline: 1681307822881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,887 DEBUG [Thread-2920] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4186 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:02,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:02,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48410 deadline: 1681307822885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,892 DEBUG [Thread-2914] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:02,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1413 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/cfc0e2072e7745f88d07fd97696fa002 2023-04-12 13:56:02,905 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1413 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/7e27e127959e4422af7ce40e526cd775 2023-04-12 13:56:02,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/a564465110f44b2e855c64c7fc380765 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a564465110f44b2e855c64c7fc380765 2023-04-12 13:56:02,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a564465110f44b2e855c64c7fc380765, entries=300, sequenceid=1413, filesize=19.0 K 2023-04-12 13:56:02,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/cfc0e2072e7745f88d07fd97696fa002 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cfc0e2072e7745f88d07fd97696fa002 2023-04-12 13:56:02,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cfc0e2072e7745f88d07fd97696fa002, entries=150, sequenceid=1413, filesize=11.9 K 2023-04-12 13:56:02,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/7e27e127959e4422af7ce40e526cd775 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/7e27e127959e4422af7ce40e526cd775 2023-04-12 13:56:02,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/7e27e127959e4422af7ce40e526cd775, entries=150, sequenceid=1413, filesize=11.9 K 2023-04-12 13:56:02,921 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.75 KB/384768, currentSize=46.96 KB/48090 for a97913894f2065758f48c5b0eb67d392 in 485ms, sequenceid=1413, compaction requested=true 2023-04-12 13:56:02,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:02,921 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:02,921 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:56:02,921 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:56:02,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:02,922 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:02,922 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54608 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:56:02,923 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47288 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:56:02,923 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:56:02,923 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:56:02,923 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,923 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,923 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5219dc0bb3a94855b4b2264f6f8fa7bc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b582e779db7d4a6f8df6da45921ea703, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d291abd9d19f4fdebc0f77f9daa38e8b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a564465110f44b2e855c64c7fc380765] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=53.3 K 2023-04-12 13:56:02,923 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35ca0617d0c0430d8d2a595e62c70786, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2be7a79b9adc4bcb93da2e66bf0ecdb4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b2a2f0a4ffe9486fbf9fe606f0b93172, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cfc0e2072e7745f88d07fd97696fa002] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=46.2 K 2023-04-12 13:56:02,924 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5219dc0bb3a94855b4b2264f6f8fa7bc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681307759834 2023-04-12 13:56:02,924 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 35ca0617d0c0430d8d2a595e62c70786, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1681307759834 2023-04-12 13:56:02,924 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b582e779db7d4a6f8df6da45921ea703, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681307760580 2023-04-12 13:56:02,924 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 2be7a79b9adc4bcb93da2e66bf0ecdb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681307760580 2023-04-12 13:56:02,925 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d291abd9d19f4fdebc0f77f9daa38e8b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=997, earliestPutTs=1681307761152 2023-04-12 13:56:02,925 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b2a2f0a4ffe9486fbf9fe606f0b93172, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=997, earliestPutTs=1681307761152 2023-04-12 13:56:02,925 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting a564465110f44b2e855c64c7fc380765, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681307762412 2023-04-12 13:56:02,925 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting cfc0e2072e7745f88d07fd97696fa002, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681307762431 2023-04-12 13:56:02,939 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:02,939 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:02,965 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:02,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:02,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:02,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:02,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:02,968 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:02,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:02,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:02,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,969 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,973 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:02,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:02,973 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,974 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:02,974 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:02,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:02,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:02,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:02,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:02,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:02,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:02,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:02,975 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:02,975 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:02,975 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:02,975 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:02,975 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:02,976 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:02,976 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:02,976 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:02,977 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:02,977 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:03,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:03,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:03,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307763111, max timestamp=1681307763165Num uniques 859; 2023-04-12 13:56:03,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307763111, max timestamp=1681307763165Num uniques 859; 2023-04-12 13:56:03,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:56:03,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:56:03,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681307763111, max timestamp=1681307763165Num uniques 859; 2023-04-12 13:56:03,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307823215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307823216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307823216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307823325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307823325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 495 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307823325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,354 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/c6e59327513d415b85baa5b1addb7d81 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c6e59327513d415b85baa5b1addb7d81 2023-04-12 13:56:03,359 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into c6e59327513d415b85baa5b1addb7d81(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:03,359 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:03,359 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=12, startTime=1681307762921; duration=0sec 2023-04-12 13:56:03,359 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:03,359 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:03,366 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/d851d7adf1b5428999a8e3547363d2c8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d851d7adf1b5428999a8e3547363d2c8 2023-04-12 13:56:03,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:03,367 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:56:03,367 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:03,367 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4f8f3e955fae4150a8959e1541a19d5f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3270fd3e08dc488ca47b06c07b618344, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/7e27e127959e4422af7ce40e526cd775] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=34.3 K 2023-04-12 13:56:03,368 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4f8f3e955fae4150a8959e1541a19d5f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=992, earliestPutTs=1681307760580 2023-04-12 13:56:03,368 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3270fd3e08dc488ca47b06c07b618344, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=997, earliestPutTs=1681307761152 2023-04-12 13:56:03,372 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7e27e127959e4422af7ce40e526cd775, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681307762431 2023-04-12 13:56:03,378 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:03,387 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into d851d7adf1b5428999a8e3547363d2c8(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:03,387 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:03,387 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=12, startTime=1681307762921; duration=0sec 2023-04-12 13:56:03,388 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:03,388 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1423 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/145ea2a5510f47b0b32cc3d30af328e6 2023-04-12 13:56:03,406 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1423 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/41c28ed411b74ec6bf8df87e79280a87 2023-04-12 13:56:03,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307823531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 497 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307823532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307823532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,798 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/77359a5bff2b49e9acec7249d81a97ea as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/77359a5bff2b49e9acec7249d81a97ea 2023-04-12 13:56:03,804 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into 77359a5bff2b49e9acec7249d81a97ea(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:03,804 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:03,804 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=13, startTime=1681307762922; duration=0sec 2023-04-12 13:56:03,804 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:03,816 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1423 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/1e2cd0837c5e4a5a90bfdedd64d9061f 2023-04-12 13:56:03,819 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/145ea2a5510f47b0b32cc3d30af328e6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/145ea2a5510f47b0b32cc3d30af328e6 2023-04-12 13:56:03,822 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/145ea2a5510f47b0b32cc3d30af328e6, entries=150, sequenceid=1423, filesize=11.9 K 2023-04-12 13:56:03,823 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/41c28ed411b74ec6bf8df87e79280a87 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/41c28ed411b74ec6bf8df87e79280a87 2023-04-12 13:56:03,826 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/41c28ed411b74ec6bf8df87e79280a87, entries=150, sequenceid=1423, filesize=11.9 K 2023-04-12 13:56:03,827 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/1e2cd0837c5e4a5a90bfdedd64d9061f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1e2cd0837c5e4a5a90bfdedd64d9061f 2023-04-12 13:56:03,837 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1e2cd0837c5e4a5a90bfdedd64d9061f, entries=150, sequenceid=1423, filesize=11.9 K 2023-04-12 13:56:03,840 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a97913894f2065758f48c5b0eb67d392 in 865ms, sequenceid=1423, compaction requested=false 2023-04-12 13:56:03,840 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:03,840 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:03,843 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,843 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:03,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:03,844 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:03,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:03,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dd5d07[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:03,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:03,845 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,847 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:03,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:03,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-12 13:56:03,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,849 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:03,849 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:03,849 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:03,849 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:03,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:03,850 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:03,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:03,850 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,850 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:03,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:03,850 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:03,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:03,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:03,854 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:03,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:03,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:03,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:03,854 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:03,857 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,857 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:03,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,857 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:03,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:03,859 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:03,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:03,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:03,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:03,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,864 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:03,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:03,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:03,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:03,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:03,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:03,864 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:03,864 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:03,864 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:03,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:03,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307823859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307823861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:03,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307823862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1463 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/fb84de8ba937429cbf41ae665daac47b 2023-04-12 13:56:03,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1463 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/de2aa181231647228236dcdf8b83dff5 2023-04-12 13:56:03,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1463 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/1af1b4bba6524828a921bbbeab148ef7 2023-04-12 13:56:03,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/fb84de8ba937429cbf41ae665daac47b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fb84de8ba937429cbf41ae665daac47b 2023-04-12 13:56:03,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fb84de8ba937429cbf41ae665daac47b, entries=250, sequenceid=1463, filesize=16.7 K 2023-04-12 13:56:03,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/de2aa181231647228236dcdf8b83dff5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/de2aa181231647228236dcdf8b83dff5 2023-04-12 13:56:03,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/de2aa181231647228236dcdf8b83dff5, entries=150, sequenceid=1463, filesize=11.9 K 2023-04-12 13:56:03,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/1af1b4bba6524828a921bbbeab148ef7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1af1b4bba6524828a921bbbeab148ef7 2023-04-12 13:56:03,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1af1b4bba6524828a921bbbeab148ef7, entries=150, sequenceid=1463, filesize=11.9 K 2023-04-12 13:56:03,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for a97913894f2065758f48c5b0eb67d392 in 103ms, sequenceid=1463, compaction requested=true 2023-04-12 13:56:03,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:03,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:03,951 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:03,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:03,951 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:03,951 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:03,953 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42562 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:03,953 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:56:03,954 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:03,954 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d851d7adf1b5428999a8e3547363d2c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/145ea2a5510f47b0b32cc3d30af328e6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fb84de8ba937429cbf41ae665daac47b] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=41.6 K 2023-04-12 13:56:03,954 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:03,954 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:56:03,954 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:03,954 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c6e59327513d415b85baa5b1addb7d81, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/41c28ed411b74ec6bf8df87e79280a87, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/de2aa181231647228236dcdf8b83dff5] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=36.8 K 2023-04-12 13:56:03,954 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d851d7adf1b5428999a8e3547363d2c8, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681307762431 2023-04-12 13:56:03,955 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c6e59327513d415b85baa5b1addb7d81, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681307762431 2023-04-12 13:56:03,955 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 145ea2a5510f47b0b32cc3d30af328e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1423, earliestPutTs=1681307762456 2023-04-12 13:56:03,955 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting fb84de8ba937429cbf41ae665daac47b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1681307763214 2023-04-12 13:56:03,955 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 41c28ed411b74ec6bf8df87e79280a87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1423, earliestPutTs=1681307762456 2023-04-12 13:56:03,956 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting de2aa181231647228236dcdf8b83dff5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1681307763215 2023-04-12 13:56:03,963 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:03,964 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:03,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:03,966 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:03,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:03,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:03,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:03,977 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:03,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:03,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:03,977 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:03,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,978 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:03,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:03,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:03,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:03,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:03,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:03,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:03,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:03,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:03,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:03,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:03,986 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:03,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:03,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:03,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:03,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:03,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:03,997 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/285711071a3f4263a3ab9c1ed9540b76 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/285711071a3f4263a3ab9c1ed9540b76 2023-04-12 13:56:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:04,020 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:04,020 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,020 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:04,022 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into 285711071a3f4263a3ab9c1ed9540b76(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:04,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:04,022 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=13, startTime=1681307763951; duration=0sec 2023-04-12 13:56:04,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:04,022 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:04,023 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:04,023 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:56:04,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1474 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b4c710467aaa4f459e331416bf9c05e9 2023-04-12 13:56:04,025 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,025 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/77359a5bff2b49e9acec7249d81a97ea, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1e2cd0837c5e4a5a90bfdedd64d9061f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1af1b4bba6524828a921bbbeab148ef7] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=36.8 K 2023-04-12 13:56:04,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:04,025 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,025 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:04,030 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:04,030 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 77359a5bff2b49e9acec7249d81a97ea, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1413, earliestPutTs=1681307762431 2023-04-12 13:56:04,031 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 1e2cd0837c5e4a5a90bfdedd64d9061f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1423, earliestPutTs=1681307762456 2023-04-12 13:56:04,032 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 1af1b4bba6524828a921bbbeab148ef7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1681307763215 2023-04-12 13:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,032 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,032 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,033 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/b140b5e466544340b63c4b6e398f1e16 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b140b5e466544340b63c4b6e398f1e16 2023-04-12 13:56:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:04,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4636f510[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:04,035 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:04,035 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,037 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,037 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:04,038 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:04,038 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,038 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:04,039 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into b140b5e466544340b63c4b6e398f1e16(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:04,040 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:04,040 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=13, startTime=1681307763951; duration=0sec 2023-04-12 13:56:04,040 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:04,040 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,040 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,041 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,045 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:04,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 511 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307824043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307764022, max timestamp=1681307764044Num uniques 859; 2023-04-12 13:56:04,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:04,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307764022, max timestamp=1681307764044Num uniques 859; 2023-04-12 13:56:04,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307764022, max timestamp=1681307764044Num uniques 859; 2023-04-12 13:56:04,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:04,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:04,048 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:04,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:04,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:04,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:04,048 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:04,050 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,050 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,050 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,051 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,052 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,054 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:04,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:04,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,055 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:04,055 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:04,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,055 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,056 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,056 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,056 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:04,086 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/998d9d1b15ec47218da854d0e3f984b1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/998d9d1b15ec47218da854d0e3f984b1 2023-04-12 13:56:04,101 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into 998d9d1b15ec47218da854d0e3f984b1(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:04,101 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:04,101 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=13, startTime=1681307763951; duration=0sec 2023-04-12 13:56:04,101 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:04,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307824120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307824121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 513 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307824147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,155 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:04,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:04,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:04,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:04,160 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:04,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:04,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:04,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,164 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,164 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:04,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:04,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,165 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:04,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:04,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:04,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:04,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:04,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:04,166 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,167 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:04,167 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:04,167 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:04,167 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:04,169 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:04,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5e80dc[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:04,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:04,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,173 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:04,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:04,173 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:04,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,175 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:04,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:04,179 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:04,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:04,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:04,179 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:04,179 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:04,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,181 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,184 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,185 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:04,188 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:04,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:04,189 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:04,189 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,189 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,189 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:04,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307824224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307824228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,289 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:04,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:04,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:04,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:04,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:04,293 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:04,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:04,293 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:04,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,295 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,295 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:04,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:04,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:04,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:04,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:04,297 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,297 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:04,297 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:04,297 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:04,297 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:04,302 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773b898e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:04,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:04,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:04,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:04,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:04,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,307 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:04,308 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:04,311 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:04,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:04,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:04,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:04,311 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:04,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,317 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:04,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:04,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:04,318 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:04,319 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:04,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 515 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307824355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,419 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:04,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:04,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:04,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:04,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:04,422 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:04,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:04,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:04,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:04,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:04,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:04,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:04,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:04,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:04,427 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,427 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:04,427 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:04,427 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:04,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:04,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:04,427 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:04,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,429 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:04,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7900d70b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:04,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:04,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,434 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:04,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:04,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:04,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 492 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307824430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:04,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:04,437 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:04,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:04,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:04,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:04,437 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:04,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307824433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,440 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,442 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:04,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,444 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:04,444 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:04,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:04,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1474 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/35d169f2d3da4005bd206e767c518d5d 2023-04-12 13:56:04,544 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:04,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:04,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:04,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:04,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:04,548 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:04,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:04,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:04,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,551 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,551 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:04,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:04,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:04,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,552 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:04,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:04,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:04,553 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:04,553 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:04,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:04,555 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,556 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:04,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86f334e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:04,557 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:04,557 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,558 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:04,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,559 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:04,562 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,562 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:04,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:04,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:04,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:04,566 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:04,566 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:04,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,569 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,570 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,572 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,575 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,575 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:04,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:04,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:04,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,576 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:04,576 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:04,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 517 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48394 deadline: 1681307824662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,676 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:04,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:04,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:04,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:04,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:04,679 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:04,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:04,679 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:04,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,680 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,681 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,681 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:04,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:04,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:04,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:04,683 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,683 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. started... 2023-04-12 13:56:04,683 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. as already flushing 2023-04-12 13:56:04,683 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:04,686 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,686 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:04,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd1aff[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:04,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:04,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:04,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:04,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:04,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:04,694 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:04,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:04,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:04,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:04,694 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:04,701 DEBUG [Thread-2933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:58527 2023-04-12 13:56:04,701 DEBUG [Thread-2933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:04,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,702 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:04,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:04,703 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,704 DEBUG [Thread-2929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:58527 2023-04-12 13:56:04,704 DEBUG [Thread-2929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:04,708 DEBUG [Thread-2927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:58527 2023-04-12 13:56:04,708 DEBUG [Thread-2927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,712 DEBUG [Thread-2925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:58527 2023-04-12 13:56:04,712 DEBUG [Thread-2925] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:04,712 DEBUG [Thread-2931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:58527 2023-04-12 13:56:04,712 DEBUG [Thread-2931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:04,713 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:04,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:04,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:04,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:04,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:04,713 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:04,713 DEBUG [Thread-2924] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:04,713 DEBUG [Thread-2924] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:04,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:04,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:04,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 494 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48372 deadline: 1681307824738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:04,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48454 deadline: 1681307824739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:04,814 DEBUG [Thread-2924] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:04,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:04,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1474 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/b2a681aca10f43bbbe0fbc1a43c7b861 2023-04-12 13:56:04,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b4c710467aaa4f459e331416bf9c05e9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b4c710467aaa4f459e331416bf9c05e9 2023-04-12 13:56:04,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b4c710467aaa4f459e331416bf9c05e9, entries=200, sequenceid=1474, filesize=14.3 K 2023-04-12 13:56:04,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/35d169f2d3da4005bd206e767c518d5d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35d169f2d3da4005bd206e767c518d5d 2023-04-12 13:56:04,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35d169f2d3da4005bd206e767c518d5d, entries=150, sequenceid=1474, filesize=11.9 K 2023-04-12 13:56:04,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/b2a681aca10f43bbbe0fbc1a43c7b861 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b2a681aca10f43bbbe0fbc1a43c7b861 2023-04-12 13:56:04,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b2a681aca10f43bbbe0fbc1a43c7b861, entries=150, sequenceid=1474, filesize=11.9 K 2023-04-12 13:56:04,903 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a97913894f2065758f48c5b0eb67d392 in 925ms, sequenceid=1474, compaction requested=false 2023-04-12 13:56:04,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:05,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:05,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 13:56:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:05,176 DEBUG [Thread-2918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:58527 2023-04-12 13:56:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:05,176 DEBUG [Thread-2918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:05,241 DEBUG [Thread-2916] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:58527 2023-04-12 13:56:05,242 DEBUG [Thread-2916] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:05,243 DEBUG [Thread-2922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:58527 2023-04-12 13:56:05,243 DEBUG [Thread-2922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:05,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1513 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/ab6eb9bd9e554582a387678855c0de9f 2023-04-12 13:56:05,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1513 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/bce5e54a92914bb48b05ff289519490b 2023-04-12 13:56:06,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1513 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/69998b3eafa44b3fa10efa80ea087e32 2023-04-12 13:56:06,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/ab6eb9bd9e554582a387678855c0de9f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab6eb9bd9e554582a387678855c0de9f 2023-04-12 13:56:06,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab6eb9bd9e554582a387678855c0de9f, entries=150, sequenceid=1513, filesize=11.9 K 2023-04-12 13:56:06,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/bce5e54a92914bb48b05ff289519490b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bce5e54a92914bb48b05ff289519490b 2023-04-12 13:56:06,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bce5e54a92914bb48b05ff289519490b, entries=150, sequenceid=1513, filesize=11.9 K 2023-04-12 13:56:06,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/69998b3eafa44b3fa10efa80ea087e32 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/69998b3eafa44b3fa10efa80ea087e32 2023-04-12 13:56:06,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/69998b3eafa44b3fa10efa80ea087e32, entries=150, sequenceid=1513, filesize=11.9 K 2023-04-12 13:56:06,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=13.42 KB/13740 for a97913894f2065758f48c5b0eb67d392 in 1239ms, sequenceid=1513, compaction requested=true 2023-04-12 13:56:06,414 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:06,414 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:06,414 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:06,414 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:06,414 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:06,414 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:06,415 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:06,415 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:06,415 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/A is initiating minor compaction (all files) 2023-04-12 13:56:06,415 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/B is initiating minor compaction (all files) 2023-04-12 13:56:06,415 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/A in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:06,415 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/B in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:06,415 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/285711071a3f4263a3ab9c1ed9540b76, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b4c710467aaa4f459e331416bf9c05e9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab6eb9bd9e554582a387678855c0de9f] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=39.3 K 2023-04-12 13:56:06,415 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b140b5e466544340b63c4b6e398f1e16, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35d169f2d3da4005bd206e767c518d5d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bce5e54a92914bb48b05ff289519490b] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=36.9 K 2023-04-12 13:56:06,416 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 285711071a3f4263a3ab9c1ed9540b76, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1681307763215 2023-04-12 13:56:06,416 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b140b5e466544340b63c4b6e398f1e16, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1681307763215 2023-04-12 13:56:06,416 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting b4c710467aaa4f459e331416bf9c05e9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1474, earliestPutTs=1681307763859 2023-04-12 13:56:06,416 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 35d169f2d3da4005bd206e767c518d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1474, earliestPutTs=1681307763860 2023-04-12 13:56:06,416 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ab6eb9bd9e554582a387678855c0de9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1513, earliestPutTs=1681307764093 2023-04-12 13:56:06,416 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bce5e54a92914bb48b05ff289519490b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1513, earliestPutTs=1681307764093 2023-04-12 13:56:06,422 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#A#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:06,424 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#B#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:06,831 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/3115c45e559f4f37bd51cebe3c95d0ab as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/3115c45e559f4f37bd51cebe3c95d0ab 2023-04-12 13:56:06,831 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/4c6b7d7d99cf4873a37ce86ed79a9158 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4c6b7d7d99cf4873a37ce86ed79a9158 2023-04-12 13:56:06,834 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/B of a97913894f2065758f48c5b0eb67d392 into 4c6b7d7d99cf4873a37ce86ed79a9158(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:06,834 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/A of a97913894f2065758f48c5b0eb67d392 into 3115c45e559f4f37bd51cebe3c95d0ab(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:06,834 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:06,835 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:06,835 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/B, priority=13, startTime=1681307766414; duration=0sec 2023-04-12 13:56:06,835 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/A, priority=13, startTime=1681307766414; duration=0sec 2023-04-12 13:56:06,835 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:06,835 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:06,835 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:06,835 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:06,836 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): a97913894f2065758f48c5b0eb67d392/C is initiating minor compaction (all files) 2023-04-12 13:56:06,836 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a97913894f2065758f48c5b0eb67d392/C in TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:06,836 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/998d9d1b15ec47218da854d0e3f984b1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b2a681aca10f43bbbe0fbc1a43c7b861, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/69998b3eafa44b3fa10efa80ea087e32] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp, totalSize=36.9 K 2023-04-12 13:56:06,836 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 998d9d1b15ec47218da854d0e3f984b1, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1463, earliestPutTs=1681307763215 2023-04-12 13:56:06,836 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b2a681aca10f43bbbe0fbc1a43c7b861, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1474, earliestPutTs=1681307763860 2023-04-12 13:56:06,836 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 69998b3eafa44b3fa10efa80ea087e32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1513, earliestPutTs=1681307764093 2023-04-12 13:56:06,841 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): a97913894f2065758f48c5b0eb67d392#C#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:06,896 DEBUG [Thread-2914] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:58527 2023-04-12 13:56:06,896 DEBUG [Thread-2914] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:06,906 DEBUG [Thread-2920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:58527 2023-04-12 13:56:06,906 DEBUG [Thread-2920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 345 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 376 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 185 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 353 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1557 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 4663 rows 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1562 2023-04-12 13:56:06,906 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 4685 rows 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1564 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 4691 rows 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1559 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 4670 rows 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(402): scanned 1553 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(403): verified 4658 rows 2023-04-12 13:56:06,907 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:56:06,907 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:58527 2023-04-12 13:56:06,907 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:06,908 INFO [Listener at localhost/46495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 13:56:06,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 13:56:06,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:06,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 13:56:06,914 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307766913"}]},"ts":"1681307766913"} 2023-04-12 13:56:06,915 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 13:56:06,917 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 13:56:06,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, UNASSIGN}] 2023-04-12 13:56:06,919 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, UNASSIGN 2023-04-12 13:56:06,920 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=a97913894f2065758f48c5b0eb67d392, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:06,920 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307766920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307766920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307766920"}]},"ts":"1681307766920"} 2023-04-12 13:56:06,921 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:56:07,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 13:56:07,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:07,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a97913894f2065758f48c5b0eb67d392, disabling compactions & flushes 2023-04-12 13:56:07,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions to complete for region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 13:56:07,251 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/b615770497c74b898e39938738eb0ee5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b615770497c74b898e39938738eb0ee5 2023-04-12 13:56:07,255 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a97913894f2065758f48c5b0eb67d392/C of a97913894f2065758f48c5b0eb67d392 into b615770497c74b898e39938738eb0ee5(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:07,255 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:07,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:07,255 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392., storeName=a97913894f2065758f48c5b0eb67d392/C, priority=13, startTime=1681307766414; duration=0sec 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:07,255 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. after waiting 0 ms 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:07,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a97913894f2065758f48c5b0eb67d392 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=A 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=B 2023-04-12 13:56:07,255 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:07,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a97913894f2065758f48c5b0eb67d392, store=C 2023-04-12 13:56:07,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 13:56:07,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1523 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b62601e036a046f0a3d8073097245baa 2023-04-12 13:56:08,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 13:56:08,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1523 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/b96072e50a9c4dfda7195b64ba45392d 2023-04-12 13:56:08,481 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1523 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/d998d8bebbb24ff1b690319b0266b49b 2023-04-12 13:56:08,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/A/b62601e036a046f0a3d8073097245baa as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b62601e036a046f0a3d8073097245baa 2023-04-12 13:56:08,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b62601e036a046f0a3d8073097245baa, entries=100, sequenceid=1523, filesize=9.5 K 2023-04-12 13:56:08,489 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/B/b96072e50a9c4dfda7195b64ba45392d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b96072e50a9c4dfda7195b64ba45392d 2023-04-12 13:56:08,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b96072e50a9c4dfda7195b64ba45392d, entries=100, sequenceid=1523, filesize=9.5 K 2023-04-12 13:56:08,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/.tmp/C/d998d8bebbb24ff1b690319b0266b49b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d998d8bebbb24ff1b690319b0266b49b 2023-04-12 13:56:08,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d998d8bebbb24ff1b690319b0266b49b, entries=100, sequenceid=1523, filesize=9.5 K 2023-04-12 13:56:08,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a97913894f2065758f48c5b0eb67d392 in 1242ms, sequenceid=1523, compaction requested=false 2023-04-12 13:56:08,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/4a9884008e4048eaa5e20f038aa4aa04, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/cb484242150e4648bae9b02192987397, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/9efefa4c53e74978ab067a3f00891d7f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/41a60127b12146979f9dba84a9384694, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c472a1449525430ea96d8d8aedf64e46, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5086e2ac8176404f9a55ad19be524473, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ea70dbbf8e9c419c9d9af89fc6741565, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ef64cea80ba44102abc530ae2c8218d4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/62bc6ec9ddde4ec59673600775d13cb3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c5a1cc0d743e495db871de8f04d972af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/51d200fe2e114742947f6827fab7bff6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c33594f41f56435188a10e4272e52c5b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/46e32bccd45143299259868996576d4a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/607620f0283b47ca8a199597282bb563, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/33c0e5c12c1f4951a8965417266b6ea3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab7e1a8705ed4532b454673c0a236803, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/eaeda299b1da42a18c59cc66e3b3fd75, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/17bbcfedea7741018bdffbfffea3db8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fa695ea0b16746d5a62e5c003ddf9b78, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a30dc3321e914924b93d568e3f3233b6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/39c148d6d1624534a76622bc04c78af5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/e9f341a309ef49e9895a8586064f77a6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/bbdf3e9fe8854b09bc353859b6113b4c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b404ea23f173404b8910fee95b7a617b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/6ef89575eedb48c5943c387399bb25e2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/78aba42b17a9466ea7823217e64565e1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c2caf69eff134092beadd8d8e03c9233, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/1c84f05104f844ffa3b9458963103339, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5219dc0bb3a94855b4b2264f6f8fa7bc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b582e779db7d4a6f8df6da45921ea703, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d291abd9d19f4fdebc0f77f9daa38e8b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a564465110f44b2e855c64c7fc380765, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d851d7adf1b5428999a8e3547363d2c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/145ea2a5510f47b0b32cc3d30af328e6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fb84de8ba937429cbf41ae665daac47b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/285711071a3f4263a3ab9c1ed9540b76, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b4c710467aaa4f459e331416bf9c05e9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab6eb9bd9e554582a387678855c0de9f] to archive 2023-04-12 13:56:08,531 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:56:08,533 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/4a9884008e4048eaa5e20f038aa4aa04 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/4a9884008e4048eaa5e20f038aa4aa04 2023-04-12 13:56:08,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/cb484242150e4648bae9b02192987397 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/cb484242150e4648bae9b02192987397 2023-04-12 13:56:08,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/9efefa4c53e74978ab067a3f00891d7f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/9efefa4c53e74978ab067a3f00891d7f 2023-04-12 13:56:08,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/41a60127b12146979f9dba84a9384694 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/41a60127b12146979f9dba84a9384694 2023-04-12 13:56:08,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c472a1449525430ea96d8d8aedf64e46 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c472a1449525430ea96d8d8aedf64e46 2023-04-12 13:56:08,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5086e2ac8176404f9a55ad19be524473 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5086e2ac8176404f9a55ad19be524473 2023-04-12 13:56:08,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ea70dbbf8e9c419c9d9af89fc6741565 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ea70dbbf8e9c419c9d9af89fc6741565 2023-04-12 13:56:08,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ef64cea80ba44102abc530ae2c8218d4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ef64cea80ba44102abc530ae2c8218d4 2023-04-12 13:56:08,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/62bc6ec9ddde4ec59673600775d13cb3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/62bc6ec9ddde4ec59673600775d13cb3 2023-04-12 13:56:08,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c5a1cc0d743e495db871de8f04d972af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c5a1cc0d743e495db871de8f04d972af 2023-04-12 13:56:08,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/51d200fe2e114742947f6827fab7bff6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/51d200fe2e114742947f6827fab7bff6 2023-04-12 13:56:08,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c33594f41f56435188a10e4272e52c5b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c33594f41f56435188a10e4272e52c5b 2023-04-12 13:56:08,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/46e32bccd45143299259868996576d4a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/46e32bccd45143299259868996576d4a 2023-04-12 13:56:08,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/888d7b8e6f2c46a1bcb6dd7ee629bd4c 2023-04-12 13:56:08,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/607620f0283b47ca8a199597282bb563 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/607620f0283b47ca8a199597282bb563 2023-04-12 13:56:08,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/33c0e5c12c1f4951a8965417266b6ea3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/33c0e5c12c1f4951a8965417266b6ea3 2023-04-12 13:56:08,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab7e1a8705ed4532b454673c0a236803 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab7e1a8705ed4532b454673c0a236803 2023-04-12 13:56:08,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/eaeda299b1da42a18c59cc66e3b3fd75 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/eaeda299b1da42a18c59cc66e3b3fd75 2023-04-12 13:56:08,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/17bbcfedea7741018bdffbfffea3db8d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/17bbcfedea7741018bdffbfffea3db8d 2023-04-12 13:56:08,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fa695ea0b16746d5a62e5c003ddf9b78 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fa695ea0b16746d5a62e5c003ddf9b78 2023-04-12 13:56:08,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a30dc3321e914924b93d568e3f3233b6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a30dc3321e914924b93d568e3f3233b6 2023-04-12 13:56:08,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/39c148d6d1624534a76622bc04c78af5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/39c148d6d1624534a76622bc04c78af5 2023-04-12 13:56:08,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/e9f341a309ef49e9895a8586064f77a6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/e9f341a309ef49e9895a8586064f77a6 2023-04-12 13:56:08,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/bbdf3e9fe8854b09bc353859b6113b4c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/bbdf3e9fe8854b09bc353859b6113b4c 2023-04-12 13:56:08,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b404ea23f173404b8910fee95b7a617b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b404ea23f173404b8910fee95b7a617b 2023-04-12 13:56:08,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/6ef89575eedb48c5943c387399bb25e2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/6ef89575eedb48c5943c387399bb25e2 2023-04-12 13:56:08,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/78aba42b17a9466ea7823217e64565e1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/78aba42b17a9466ea7823217e64565e1 2023-04-12 13:56:08,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c2caf69eff134092beadd8d8e03c9233 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/c2caf69eff134092beadd8d8e03c9233 2023-04-12 13:56:08,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/1c84f05104f844ffa3b9458963103339 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/1c84f05104f844ffa3b9458963103339 2023-04-12 13:56:08,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5219dc0bb3a94855b4b2264f6f8fa7bc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/5219dc0bb3a94855b4b2264f6f8fa7bc 2023-04-12 13:56:08,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b582e779db7d4a6f8df6da45921ea703 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b582e779db7d4a6f8df6da45921ea703 2023-04-12 13:56:08,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d291abd9d19f4fdebc0f77f9daa38e8b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d291abd9d19f4fdebc0f77f9daa38e8b 2023-04-12 13:56:08,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a564465110f44b2e855c64c7fc380765 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/a564465110f44b2e855c64c7fc380765 2023-04-12 13:56:08,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d851d7adf1b5428999a8e3547363d2c8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/d851d7adf1b5428999a8e3547363d2c8 2023-04-12 13:56:08,561 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/145ea2a5510f47b0b32cc3d30af328e6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/145ea2a5510f47b0b32cc3d30af328e6 2023-04-12 13:56:08,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fb84de8ba937429cbf41ae665daac47b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/fb84de8ba937429cbf41ae665daac47b 2023-04-12 13:56:08,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/285711071a3f4263a3ab9c1ed9540b76 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/285711071a3f4263a3ab9c1ed9540b76 2023-04-12 13:56:08,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b4c710467aaa4f459e331416bf9c05e9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b4c710467aaa4f459e331416bf9c05e9 2023-04-12 13:56:08,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab6eb9bd9e554582a387678855c0de9f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/ab6eb9bd9e554582a387678855c0de9f 2023-04-12 13:56:08,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/15e6517df18647b281d72632a254dc78, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/97abe845bbb44814b6428ae6d0faa09f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/84862e704ceb4f048301b748ddd717bc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2bc2b328672649b4a53e0e427b634e58, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/20537c07b4d8405681f39759dd09e08b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/93109353cc694cb387fbcf8003720104, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bcf9a6840d7640b280b887702b66c968, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4106650dfda348b58e957faad2648112, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/5d164e070012416da933912cbf02806e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4f8985ac1bc94733a5b716c61adce043, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/af1c7882dd3548ba874a073a47df805b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/9ad8609c07cd4780831303c5a8d7d1b8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/666fadebdfb6458d8056208560ba509b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/0d09670edc2b4022898b1be879a9b362, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cab1ffe5cc91492ab870a5fb4a09d51b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/1cf1913ce2d54ddf92713ccf58975a2d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/36147211179c494bb3572082898426ca, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4aae9729427b45c29d3d1942ff7c403d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eec945e727ab4bad90b63af494bbf2aa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a9700e755d1040abb6595d90cd409f35, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/42bb2cd1272145c4b1b94b485cbffaab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/f1f52ab2e3f64940bafc1acd99adc56a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/e82cb676a24f4ae2a6e1a5ce03189a19, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a4096c2c71fd40d6b44e23a5252f21fe, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/34fe39d5533f4786b266cf0780a4308d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eaa7cc36915d4018bb4b8e6f5d3477d8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c98970ab0105476987ef7b4649a3dacc, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/640f748fd6c546e3abe64cf9475e352c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35ca0617d0c0430d8d2a595e62c70786, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/d7acadd2ab664b62bbac5947e0556fa8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2be7a79b9adc4bcb93da2e66bf0ecdb4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b2a2f0a4ffe9486fbf9fe606f0b93172, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c6e59327513d415b85baa5b1addb7d81, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cfc0e2072e7745f88d07fd97696fa002, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/41c28ed411b74ec6bf8df87e79280a87, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b140b5e466544340b63c4b6e398f1e16, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/de2aa181231647228236dcdf8b83dff5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35d169f2d3da4005bd206e767c518d5d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bce5e54a92914bb48b05ff289519490b] to archive 2023-04-12 13:56:08,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:56:08,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/15e6517df18647b281d72632a254dc78 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/15e6517df18647b281d72632a254dc78 2023-04-12 13:56:08,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/97abe845bbb44814b6428ae6d0faa09f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/97abe845bbb44814b6428ae6d0faa09f 2023-04-12 13:56:08,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/84862e704ceb4f048301b748ddd717bc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/84862e704ceb4f048301b748ddd717bc 2023-04-12 13:56:08,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2bc2b328672649b4a53e0e427b634e58 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2bc2b328672649b4a53e0e427b634e58 2023-04-12 13:56:08,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/20537c07b4d8405681f39759dd09e08b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/20537c07b4d8405681f39759dd09e08b 2023-04-12 13:56:08,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/93109353cc694cb387fbcf8003720104 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/93109353cc694cb387fbcf8003720104 2023-04-12 13:56:08,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bcf9a6840d7640b280b887702b66c968 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bcf9a6840d7640b280b887702b66c968 2023-04-12 13:56:08,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4106650dfda348b58e957faad2648112 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4106650dfda348b58e957faad2648112 2023-04-12 13:56:08,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/5d164e070012416da933912cbf02806e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/5d164e070012416da933912cbf02806e 2023-04-12 13:56:08,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4f8985ac1bc94733a5b716c61adce043 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4f8985ac1bc94733a5b716c61adce043 2023-04-12 13:56:08,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/af1c7882dd3548ba874a073a47df805b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/af1c7882dd3548ba874a073a47df805b 2023-04-12 13:56:08,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/9ad8609c07cd4780831303c5a8d7d1b8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/9ad8609c07cd4780831303c5a8d7d1b8 2023-04-12 13:56:08,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/666fadebdfb6458d8056208560ba509b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/666fadebdfb6458d8056208560ba509b 2023-04-12 13:56:08,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/0d09670edc2b4022898b1be879a9b362 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/0d09670edc2b4022898b1be879a9b362 2023-04-12 13:56:08,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cab1ffe5cc91492ab870a5fb4a09d51b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cab1ffe5cc91492ab870a5fb4a09d51b 2023-04-12 13:56:08,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/1cf1913ce2d54ddf92713ccf58975a2d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/1cf1913ce2d54ddf92713ccf58975a2d 2023-04-12 13:56:08,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/36147211179c494bb3572082898426ca to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/36147211179c494bb3572082898426ca 2023-04-12 13:56:08,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4aae9729427b45c29d3d1942ff7c403d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4aae9729427b45c29d3d1942ff7c403d 2023-04-12 13:56:08,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eec945e727ab4bad90b63af494bbf2aa to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eec945e727ab4bad90b63af494bbf2aa 2023-04-12 13:56:08,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a9700e755d1040abb6595d90cd409f35 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a9700e755d1040abb6595d90cd409f35 2023-04-12 13:56:08,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/42bb2cd1272145c4b1b94b485cbffaab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/42bb2cd1272145c4b1b94b485cbffaab 2023-04-12 13:56:08,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/f1f52ab2e3f64940bafc1acd99adc56a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/f1f52ab2e3f64940bafc1acd99adc56a 2023-04-12 13:56:08,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/e82cb676a24f4ae2a6e1a5ce03189a19 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/e82cb676a24f4ae2a6e1a5ce03189a19 2023-04-12 13:56:08,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a4096c2c71fd40d6b44e23a5252f21fe to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/a4096c2c71fd40d6b44e23a5252f21fe 2023-04-12 13:56:08,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/34fe39d5533f4786b266cf0780a4308d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/34fe39d5533f4786b266cf0780a4308d 2023-04-12 13:56:08,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eaa7cc36915d4018bb4b8e6f5d3477d8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/eaa7cc36915d4018bb4b8e6f5d3477d8 2023-04-12 13:56:08,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c98970ab0105476987ef7b4649a3dacc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c98970ab0105476987ef7b4649a3dacc 2023-04-12 13:56:08,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/640f748fd6c546e3abe64cf9475e352c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/640f748fd6c546e3abe64cf9475e352c 2023-04-12 13:56:08,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35ca0617d0c0430d8d2a595e62c70786 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35ca0617d0c0430d8d2a595e62c70786 2023-04-12 13:56:08,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/d7acadd2ab664b62bbac5947e0556fa8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/d7acadd2ab664b62bbac5947e0556fa8 2023-04-12 13:56:08,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2be7a79b9adc4bcb93da2e66bf0ecdb4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/2be7a79b9adc4bcb93da2e66bf0ecdb4 2023-04-12 13:56:08,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b2a2f0a4ffe9486fbf9fe606f0b93172 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b2a2f0a4ffe9486fbf9fe606f0b93172 2023-04-12 13:56:08,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c6e59327513d415b85baa5b1addb7d81 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/c6e59327513d415b85baa5b1addb7d81 2023-04-12 13:56:08,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cfc0e2072e7745f88d07fd97696fa002 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/cfc0e2072e7745f88d07fd97696fa002 2023-04-12 13:56:08,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/41c28ed411b74ec6bf8df87e79280a87 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/41c28ed411b74ec6bf8df87e79280a87 2023-04-12 13:56:08,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b140b5e466544340b63c4b6e398f1e16 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b140b5e466544340b63c4b6e398f1e16 2023-04-12 13:56:08,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/de2aa181231647228236dcdf8b83dff5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/de2aa181231647228236dcdf8b83dff5 2023-04-12 13:56:08,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35d169f2d3da4005bd206e767c518d5d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/35d169f2d3da4005bd206e767c518d5d 2023-04-12 13:56:08,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bce5e54a92914bb48b05ff289519490b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/bce5e54a92914bb48b05ff289519490b 2023-04-12 13:56:08,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/072d39290f0b459f819e9f150a66c55a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/caf6e284f97c46b093161208cca694b1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d48e814ec1af46ec88e4c55513b99fab, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8aa8de00dbf04c52825f3a732a5ff7a3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4cf3eef06f99484dacf265abfc7b0f84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/590091183bfa478d8dbdb19f844acd59, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2702a9f2332d4482ac88f3e3ccc94bee, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/5b06b002ab084413a7ddcc2a03ddb165, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/294d5fed36e84feb9264bc91bd6af2ba, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/45fefa277daf4b2787d6d0ffc7b0a4e4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/6d14c56b9b944f9987f736a678b5c71c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0e96d24e88f34e41b7129190abe0e1e8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8b6dd3ce4f9d4cdebad8448800393a38, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2b4a63323598459b95b443514af84170, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3cf093efa3aa4936812ecef35e117038, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f3af831368844c7a9b171106ab302e84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/68b62e98eb8241d6bf6478da279310d4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48317a2481e64c14bd5954245f337bb2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b9d942e63e3e4ccf81983770a273c0fe, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67403f25746e458fa6b248dc0d8b9f6c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0188a5f380d14d1c9b77badf3a5b76aa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f631e4bc92294062a0eebb879124c889, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67c6a172b2414b3292dd83aac5f34d51, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0089e1adf9f54a29b6bef2bd1303f6a7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/e63180c5580b418597b04b5afd22bf36, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/093e21f26a7047b681a5a41f1dd199d0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/ebf23db91b5f496e89b45d04a02eddac, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/fc4968080509482c87c429f32ff3ed89, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48d4440ea9ee449ea25f8d40d1cbbd53, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4f8f3e955fae4150a8959e1541a19d5f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f11b1e7a6c874d98a857df6df2cc6366, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3270fd3e08dc488ca47b06c07b618344, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/77359a5bff2b49e9acec7249d81a97ea, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/7e27e127959e4422af7ce40e526cd775, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1e2cd0837c5e4a5a90bfdedd64d9061f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/998d9d1b15ec47218da854d0e3f984b1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1af1b4bba6524828a921bbbeab148ef7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b2a681aca10f43bbbe0fbc1a43c7b861, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/69998b3eafa44b3fa10efa80ea087e32] to archive 2023-04-12 13:56:08,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:56:08,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/072d39290f0b459f819e9f150a66c55a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/072d39290f0b459f819e9f150a66c55a 2023-04-12 13:56:08,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/caf6e284f97c46b093161208cca694b1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/caf6e284f97c46b093161208cca694b1 2023-04-12 13:56:08,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d48e814ec1af46ec88e4c55513b99fab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d48e814ec1af46ec88e4c55513b99fab 2023-04-12 13:56:08,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8aa8de00dbf04c52825f3a732a5ff7a3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8aa8de00dbf04c52825f3a732a5ff7a3 2023-04-12 13:56:08,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4cf3eef06f99484dacf265abfc7b0f84 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4cf3eef06f99484dacf265abfc7b0f84 2023-04-12 13:56:08,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/590091183bfa478d8dbdb19f844acd59 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/590091183bfa478d8dbdb19f844acd59 2023-04-12 13:56:08,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2702a9f2332d4482ac88f3e3ccc94bee to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2702a9f2332d4482ac88f3e3ccc94bee 2023-04-12 13:56:08,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/5b06b002ab084413a7ddcc2a03ddb165 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/5b06b002ab084413a7ddcc2a03ddb165 2023-04-12 13:56:08,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/294d5fed36e84feb9264bc91bd6af2ba to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/294d5fed36e84feb9264bc91bd6af2ba 2023-04-12 13:56:08,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/45fefa277daf4b2787d6d0ffc7b0a4e4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/45fefa277daf4b2787d6d0ffc7b0a4e4 2023-04-12 13:56:08,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/6d14c56b9b944f9987f736a678b5c71c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/6d14c56b9b944f9987f736a678b5c71c 2023-04-12 13:56:08,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0e96d24e88f34e41b7129190abe0e1e8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0e96d24e88f34e41b7129190abe0e1e8 2023-04-12 13:56:08,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8b6dd3ce4f9d4cdebad8448800393a38 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/8b6dd3ce4f9d4cdebad8448800393a38 2023-04-12 13:56:08,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2b4a63323598459b95b443514af84170 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/2b4a63323598459b95b443514af84170 2023-04-12 13:56:08,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3cf093efa3aa4936812ecef35e117038 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3cf093efa3aa4936812ecef35e117038 2023-04-12 13:56:08,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f3af831368844c7a9b171106ab302e84 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f3af831368844c7a9b171106ab302e84 2023-04-12 13:56:08,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/68b62e98eb8241d6bf6478da279310d4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/68b62e98eb8241d6bf6478da279310d4 2023-04-12 13:56:08,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48317a2481e64c14bd5954245f337bb2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48317a2481e64c14bd5954245f337bb2 2023-04-12 13:56:08,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b9d942e63e3e4ccf81983770a273c0fe to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b9d942e63e3e4ccf81983770a273c0fe 2023-04-12 13:56:08,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67403f25746e458fa6b248dc0d8b9f6c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67403f25746e458fa6b248dc0d8b9f6c 2023-04-12 13:56:08,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0188a5f380d14d1c9b77badf3a5b76aa to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0188a5f380d14d1c9b77badf3a5b76aa 2023-04-12 13:56:08,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f631e4bc92294062a0eebb879124c889 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f631e4bc92294062a0eebb879124c889 2023-04-12 13:56:08,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67c6a172b2414b3292dd83aac5f34d51 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/67c6a172b2414b3292dd83aac5f34d51 2023-04-12 13:56:08,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0089e1adf9f54a29b6bef2bd1303f6a7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/0089e1adf9f54a29b6bef2bd1303f6a7 2023-04-12 13:56:08,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/e63180c5580b418597b04b5afd22bf36 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/e63180c5580b418597b04b5afd22bf36 2023-04-12 13:56:08,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/093e21f26a7047b681a5a41f1dd199d0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/093e21f26a7047b681a5a41f1dd199d0 2023-04-12 13:56:08,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/ebf23db91b5f496e89b45d04a02eddac to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/ebf23db91b5f496e89b45d04a02eddac 2023-04-12 13:56:08,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/fc4968080509482c87c429f32ff3ed89 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/fc4968080509482c87c429f32ff3ed89 2023-04-12 13:56:08,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48d4440ea9ee449ea25f8d40d1cbbd53 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/48d4440ea9ee449ea25f8d40d1cbbd53 2023-04-12 13:56:08,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4f8f3e955fae4150a8959e1541a19d5f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/4f8f3e955fae4150a8959e1541a19d5f 2023-04-12 13:56:08,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f11b1e7a6c874d98a857df6df2cc6366 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/f11b1e7a6c874d98a857df6df2cc6366 2023-04-12 13:56:08,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3270fd3e08dc488ca47b06c07b618344 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/3270fd3e08dc488ca47b06c07b618344 2023-04-12 13:56:08,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/77359a5bff2b49e9acec7249d81a97ea to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/77359a5bff2b49e9acec7249d81a97ea 2023-04-12 13:56:08,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/7e27e127959e4422af7ce40e526cd775 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/7e27e127959e4422af7ce40e526cd775 2023-04-12 13:56:08,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1e2cd0837c5e4a5a90bfdedd64d9061f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1e2cd0837c5e4a5a90bfdedd64d9061f 2023-04-12 13:56:08,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/998d9d1b15ec47218da854d0e3f984b1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/998d9d1b15ec47218da854d0e3f984b1 2023-04-12 13:56:08,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1af1b4bba6524828a921bbbeab148ef7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/1af1b4bba6524828a921bbbeab148ef7 2023-04-12 13:56:08,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b2a681aca10f43bbbe0fbc1a43c7b861 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b2a681aca10f43bbbe0fbc1a43c7b861 2023-04-12 13:56:08,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/69998b3eafa44b3fa10efa80ea087e32 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/69998b3eafa44b3fa10efa80ea087e32 2023-04-12 13:56:08,642 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/recovered.edits/1526.seqid, newMaxSeqId=1526, maxSeqId=1 2023-04-12 13:56:08,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392. 2023-04-12 13:56:08,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a97913894f2065758f48c5b0eb67d392: 2023-04-12 13:56:08,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:08,646 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=a97913894f2065758f48c5b0eb67d392, regionState=CLOSED 2023-04-12 13:56:08,646 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307768646"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307768646"}]},"ts":"1681307768646"} 2023-04-12 13:56:08,648 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-12 13:56:08,648 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure a97913894f2065758f48c5b0eb67d392, server=jenkins-hbase4.apache.org,35663,1681307616979 in 1.7260 sec 2023-04-12 13:56:08,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-12 13:56:08,649 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a97913894f2065758f48c5b0eb67d392, UNASSIGN in 1.7300 sec 2023-04-12 13:56:08,649 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307768649"}]},"ts":"1681307768649"} 2023-04-12 13:56:08,650 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 13:56:08,652 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 13:56:08,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7430 sec 2023-04-12 13:56:09,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-12 13:56:09,018 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-12 13:56:09,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 13:56:09,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,021 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-12 13:56:09,022 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,025 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:09,027 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/recovered.edits] 2023-04-12 13:56:09,029 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/3115c45e559f4f37bd51cebe3c95d0ab to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/3115c45e559f4f37bd51cebe3c95d0ab 2023-04-12 13:56:09,030 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b62601e036a046f0a3d8073097245baa to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/A/b62601e036a046f0a3d8073097245baa 2023-04-12 13:56:09,032 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4c6b7d7d99cf4873a37ce86ed79a9158 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/4c6b7d7d99cf4873a37ce86ed79a9158 2023-04-12 13:56:09,033 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b96072e50a9c4dfda7195b64ba45392d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/B/b96072e50a9c4dfda7195b64ba45392d 2023-04-12 13:56:09,035 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b615770497c74b898e39938738eb0ee5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/b615770497c74b898e39938738eb0ee5 2023-04-12 13:56:09,036 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d998d8bebbb24ff1b690319b0266b49b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/C/d998d8bebbb24ff1b690319b0266b49b 2023-04-12 13:56:09,039 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/recovered.edits/1526.seqid to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392/recovered.edits/1526.seqid 2023-04-12 13:56:09,039 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/a97913894f2065758f48c5b0eb67d392 2023-04-12 13:56:09,039 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:56:09,042 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,043 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 13:56:09,045 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 13:56:09,046 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,046 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 13:56:09,046 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681307769046"}]},"ts":"9223372036854775807"} 2023-04-12 13:56:09,048 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 13:56:09,048 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a97913894f2065758f48c5b0eb67d392, NAME => 'TestAcidGuarantees,,1681307743523.a97913894f2065758f48c5b0eb67d392.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 13:56:09,048 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 13:56:09,048 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681307769048"}]},"ts":"9223372036854775807"} 2023-04-12 13:56:09,049 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 13:56:09,051 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,052 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-12 13:56:09,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-12 13:56:09,123 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-12 13:56:09,134 INFO [Listener at localhost/46495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=229 (was 229), OpenFileDescriptor=572 (was 524) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=415 (was 385) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=772 (was 633) - AvailableMemoryMB LEAK? - 2023-04-12 13:56:09,146 INFO [Listener at localhost/46495] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=229, OpenFileDescriptor=572, MaxFileDescriptor=60000, SystemLoadAverage=415, ProcessCount=171, AvailableMemoryMB=771 2023-04-12 13:56:09,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:56:09,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:56:09,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:09,151 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 13:56:09,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-12 13:56:09,152 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 13:56:09,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 13:56:09,153 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:09,153 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 empty. 2023-04-12 13:56:09,153 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:09,153 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:56:09,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 13:56:09,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 13:56:09,560 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:56:09,561 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e35088e2f9a6947abbfc447493169cd6, NAME => 'TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp 2023-04-12 13:56:09,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 13:56:09,976 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:56:09,977 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e35088e2f9a6947abbfc447493169cd6, disabling compactions & flushes 2023-04-12 13:56:09,977 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:09,977 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:09,977 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. after waiting 0 ms 2023-04-12 13:56:09,977 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:09,977 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:09,977 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:09,979 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 13:56:09,980 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307769980"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307769980"}]},"ts":"1681307769980"} 2023-04-12 13:56:09,981 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 13:56:09,982 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 13:56:09,982 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307769982"}]},"ts":"1681307769982"} 2023-04-12 13:56:09,983 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-12 13:56:09,986 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, ASSIGN}] 2023-04-12 13:56:09,987 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, ASSIGN 2023-04-12 13:56:09,988 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=false 2023-04-12 13:56:10,139 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:10,139 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307770139"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307770139"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307770139"}]},"ts":"1681307770139"} 2023-04-12 13:56:10,140 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:56:10,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 13:56:10,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:10,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e35088e2f9a6947abbfc447493169cd6, NAME => 'TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:56:10,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:56:10,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,297 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,299 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A 2023-04-12 13:56:10,299 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A 2023-04-12 13:56:10,299 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:56:10,299 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e35088e2f9a6947abbfc447493169cd6 columnFamilyName A 2023-04-12 13:56:10,300 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(310): Store=e35088e2f9a6947abbfc447493169cd6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:56:10,300 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,301 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B 2023-04-12 13:56:10,301 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B 2023-04-12 13:56:10,301 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:56:10,301 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e35088e2f9a6947abbfc447493169cd6 columnFamilyName B 2023-04-12 13:56:10,302 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(310): Store=e35088e2f9a6947abbfc447493169cd6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:56:10,302 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,303 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C 2023-04-12 13:56:10,303 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C 2023-04-12 13:56:10,303 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:56:10,303 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e35088e2f9a6947abbfc447493169cd6 columnFamilyName C 2023-04-12 13:56:10,304 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(310): Store=e35088e2f9a6947abbfc447493169cd6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:56:10,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:10,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:56:10,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:10,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 13:56:10,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e35088e2f9a6947abbfc447493169cd6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71101284, jitterRate=0.05949169397354126}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:56:10,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:10,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., pid=56, masterSystemTime=1681307770292 2023-04-12 13:56:10,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:10,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:10,312 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:10,312 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307770312"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307770312"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307770312"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307770312"}]},"ts":"1681307770312"} 2023-04-12 13:56:10,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-12 13:56:10,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 in 173 msec 2023-04-12 13:56:10,316 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-12 13:56:10,316 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, ASSIGN in 328 msec 2023-04-12 13:56:10,316 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 13:56:10,316 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307770316"}]},"ts":"1681307770316"} 2023-04-12 13:56:10,317 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-12 13:56:10,319 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 13:56:10,320 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-04-12 13:56:11,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-12 13:56:11,257 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-12 13:56:11,258 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,264 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,265 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,266 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,267 DEBUG [Listener at localhost/46495] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 13:56:11,269 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 13:56:11,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-12 13:56:11,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 13:56:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:11,282 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 13:56:11,283 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-12 13:56:11,283 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-12 13:56:11,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-12 13:56:11,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, REOPEN/MOVE}] 2023-04-12 13:56:11,288 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, REOPEN/MOVE 2023-04-12 13:56:11,288 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,288 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307771288"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307771288"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307771288"}]},"ts":"1681307771288"} 2023-04-12 13:56:11,289 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:56:11,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e35088e2f9a6947abbfc447493169cd6, disabling compactions & flushes 2023-04-12 13:56:11,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. after waiting 0 ms 2023-04-12 13:56:11,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 13:56:11,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:11,446 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e35088e2f9a6947abbfc447493169cd6 to self. 2023-04-12 13:56:11,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,448 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=CLOSED 2023-04-12 13:56:11,448 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307771448"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307771448"}]},"ts":"1681307771448"} 2023-04-12 13:56:11,450 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-12 13:56:11,450 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 in 160 msec 2023-04-12 13:56:11,451 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35663,1681307616979; forceNewPlan=false, retain=true 2023-04-12 13:56:11,601 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,601 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307771601"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307771601"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307771601"}]},"ts":"1681307771601"} 2023-04-12 13:56:11,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:56:11,756 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e35088e2f9a6947abbfc447493169cd6, NAME => 'TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.', STARTKEY => '', ENDKEY => ''} 2023-04-12 13:56:11,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 13:56:11,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,758 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,759 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A 2023-04-12 13:56:11,759 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A 2023-04-12 13:56:11,759 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:56:11,760 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e35088e2f9a6947abbfc447493169cd6 columnFamilyName A 2023-04-12 13:56:11,761 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(310): Store=e35088e2f9a6947abbfc447493169cd6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:56:11,761 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,761 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B 2023-04-12 13:56:11,761 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B 2023-04-12 13:56:11,762 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:56:11,762 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e35088e2f9a6947abbfc447493169cd6 columnFamilyName B 2023-04-12 13:56:11,762 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(310): Store=e35088e2f9a6947abbfc447493169cd6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:56:11,762 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,763 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C 2023-04-12 13:56:11,763 DEBUG [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C 2023-04-12 13:56:11,763 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-12 13:56:11,763 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e35088e2f9a6947abbfc447493169cd6 columnFamilyName C 2023-04-12 13:56:11,763 INFO [StoreOpener-e35088e2f9a6947abbfc447493169cd6-1] regionserver.HStore(310): Store=e35088e2f9a6947abbfc447493169cd6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 13:56:11,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-12 13:56:11,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e35088e2f9a6947abbfc447493169cd6; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74002088, jitterRate=0.10271704196929932}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-12 13:56:11,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:11,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., pid=61, masterSystemTime=1681307771754 2023-04-12 13:56:11,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,771 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,771 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307771771"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681307771771"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681307771771"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681307771771"}]},"ts":"1681307771771"} 2023-04-12 13:56:11,773 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-12 13:56:11,773 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 in 170 msec 2023-04-12 13:56:11,774 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-12 13:56:11,774 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, REOPEN/MOVE in 486 msec 2023-04-12 13:56:11,775 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-12 13:56:11,775 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-12 13:56:11,777 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 505 msec 2023-04-12 13:56:11,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-12 13:56:11,778 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,783 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,783 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,787 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,788 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,794 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,795 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,800 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,800 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x06799455 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,804 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25baee3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,804 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,808 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,809 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,815 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,815 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,821 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,822 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,826 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,826 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(139): Connect 0x28164190 to 127.0.0.1:58527 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 13:56:11,830 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d52321, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 13:56:11,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:11,842 DEBUG [hconnection-0x4164cebe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,842 DEBUG [hconnection-0x694ffec5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,845 DEBUG [hconnection-0x114b0889-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,845 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44752, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,849 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:11,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:11,849 DEBUG [hconnection-0x1115d20a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,849 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,849 DEBUG [hconnection-0x705126d4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,849 DEBUG [hconnection-0x9c18b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,849 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:11,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:11,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:11,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:11,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:11,850 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:11,850 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,853 DEBUG [hconnection-0x7e4ef054-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,853 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,853 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44774, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,853 DEBUG [hconnection-0x684744b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,853 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,854 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:11,854 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:11,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:11,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:11,855 DEBUG [hconnection-0x4ed0df92-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,855 DEBUG [hconnection-0x5d9296e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 13:56:11,857 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:11,857 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,857 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:11,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:11,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:11,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:11,861 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:11,861 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 13:56:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:11,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:11,867 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:11,870 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-12 13:56:11,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:11,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:11,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:11,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:11,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:11,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:11,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:11,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:11,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681307771876, max timestamp=1681307771883Num uniques 859; 2023-04-12 13:56:11,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-12 13:56:11,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307771876, max timestamp=1681307771883Num uniques 859; 2023-04-12 13:56:11,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:11,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307771876, max timestamp=1681307771883Num uniques 859; 2023-04-12 13:56:11,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:11,897 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304125476da05cefb4450a85e8abf847e2b8e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125476da05cefb4450a85e8abf847e2b8e 2023-04-12 13:56:11,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307771892, max timestamp=1681307771901Num uniques 1118; 2023-04-12 13:56:11,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-12 13:56:11,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681307771892, max timestamp=1681307771901Num uniques 1068; 2023-04-12 13:56:11,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-12 13:56:11,905 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=11, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c663ddbb2ad44c58920bfc0b70aa20c9 2023-04-12 13:56:11,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307771892, max timestamp=1681307771901Num uniques 1068; 2023-04-12 13:56:11,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-12 13:56:11,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:11,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307831961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:11,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307831963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:11,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307831964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:11,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8094, totalHeapSize=18648, min timestamp=1681307771935, max timestamp=1681307771964Num uniques 1077; 2023-04-12 13:56:11,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49314, new segment data size=8094, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-12 13:56:11,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681307771935, max timestamp=1681307771964Num uniques 1009; 2023-04-12 13:56:11,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-12 13:56:11,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307771935, max timestamp=1681307771964Num uniques 1077; 2023-04-12 13:56:11,968 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=6870, suffix heap size=139976,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=150 2023-04-12 13:56:11,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681307771977, max timestamp=1681307771991Num uniques 1009; 2023-04-12 13:56:11,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681307771977, max timestamp=1681307771991Num uniques 1036; 2023-04-12 13:56:11,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681307771977, max timestamp=1681307771991Num uniques 1036; 2023-04-12 13:56:11,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-12 13:56:11,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-12 13:56:11,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-12 13:56:12,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1681307772010, max timestamp=1681307772017Num uniques 1095; 2023-04-12 13:56:12,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1681307772010, max timestamp=1681307772017Num uniques 1095; 2023-04-12 13:56:12,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=8920, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-12 13:56:12,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=8920, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-12 13:56:12,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681307772010, max timestamp=1681307772016Num uniques 1095; 2023-04-12 13:56:12,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-12 13:56:12,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307772047, max timestamp=1681307772058Num uniques 1054; 2023-04-12 13:56:12,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 13:56:12,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307772047, max timestamp=1681307772058Num uniques 1054; 2023-04-12 13:56:12,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 13:56:12,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681307772047, max timestamp=1681307772058Num uniques 1054; 2023-04-12 13:56:12,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-12 13:56:12,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307772074, max timestamp=1681307772078Num uniques 1013; 2023-04-12 13:56:12,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307772074, max timestamp=1681307772078Num uniques 1013; 2023-04-12 13:56:12,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-12 13:56:12,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-12 13:56:12,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681307772074, max timestamp=1681307772078Num uniques 1013; 2023-04-12 13:56:12,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-12 13:56:12,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681307772092, max timestamp=1681307772097Num uniques 1022; 2023-04-12 13:56:12,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-12 13:56:12,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681307772092, max timestamp=1681307772097Num uniques 1022; 2023-04-12 13:56:12,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-12 13:56:12,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681307772092, max timestamp=1681307772097Num uniques 1022; 2023-04-12 13:56:12,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-12 13:56:12,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307832110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307832110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307832111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681307772104, max timestamp=1681307772112Num uniques 1081; 2023-04-12 13:56:12,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-12 13:56:12,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681307772104, max timestamp=1681307772112Num uniques 1081; 2023-04-12 13:56:12,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-12 13:56:12,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681307772104, max timestamp=1681307772112Num uniques 1081; 2023-04-12 13:56:12,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-12 13:56:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681307772125, max timestamp=1681307772138Num uniques 1040; 2023-04-12 13:56:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-12 13:56:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681307772125, max timestamp=1681307772138Num uniques 1040; 2023-04-12 13:56:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681307772125, max timestamp=1681307772138Num uniques 1040; 2023-04-12 13:56:12,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-12 13:56:12,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-12 13:56:12,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681307772155, max timestamp=1681307772164Num uniques 1099; 2023-04-12 13:56:12,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-12 13:56:12,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681307772155, max timestamp=1681307772164Num uniques 1099; 2023-04-12 13:56:12,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-12 13:56:12,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681307772155, max timestamp=1681307772164Num uniques 1099; 2023-04-12 13:56:12,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-12 13:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681307772180, max timestamp=1681307772196Num uniques 1058; 2023-04-12 13:56:12,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-12 13:56:12,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307772180, max timestamp=1681307772196Num uniques 1058; 2023-04-12 13:56:12,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=6870, suffix heap size=138000,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=150 2023-04-12 13:56:12,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307772180, max timestamp=1681307772196Num uniques 1058; 2023-04-12 13:56:12,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=6870, suffix heap size=138000,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=150 2023-04-12 13:56:12,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681307772211, max timestamp=1681307772217Num uniques 1009; 2023-04-12 13:56:12,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681307772211, max timestamp=1681307772217Num uniques 1017; 2023-04-12 13:56:12,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681307772211, max timestamp=1681307772217Num uniques 1009; 2023-04-12 13:56:12,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=9932, suffix heap size=132904,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=217 2023-04-12 13:56:12,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=9932, suffix heap size=132904,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=217 2023-04-12 13:56:12,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-12 13:56:12,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307772225, max timestamp=1681307772253Num uniques 1076; 2023-04-12 13:56:12,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-12 13:56:12,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307772225, max timestamp=1681307772253Num uniques 1076; 2023-04-12 13:56:12,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-12 13:56:12,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681307772225, max timestamp=1681307772253Num uniques 1076; 2023-04-12 13:56:12,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-12 13:56:12,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681307772259, max timestamp=1681307772269Num uniques 1035; 2023-04-12 13:56:12,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-12 13:56:12,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681307772259, max timestamp=1681307772269Num uniques 1035; 2023-04-12 13:56:12,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-12 13:56:12,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681307772259, max timestamp=1681307772269Num uniques 1035; 2023-04-12 13:56:12,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-12 13:56:12,295 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681307772277, max timestamp=1681307772293Num uniques 1094; 2023-04-12 13:56:12,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681307772277, max timestamp=1681307772293Num uniques 1094; 2023-04-12 13:56:12,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-12 13:56:12,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-12 13:56:12,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681307772277, max timestamp=1681307772293Num uniques 1094; 2023-04-12 13:56:12,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-12 13:56:12,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681307772302, max timestamp=1681307772309Num uniques 1203; 2023-04-12 13:56:12,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=9295, suffix heap size=153080,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=203 2023-04-12 13:56:12,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681307772302, max timestamp=1681307772309Num uniques 1203; 2023-04-12 13:56:12,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=9295, suffix heap size=153080,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=203 2023-04-12 13:56:12,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681307772302, max timestamp=1681307772309Num uniques 1203; 2023-04-12 13:56:12,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=55095, new segment data size=9295, suffix heap size=153080,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1203, new segment cells count=203 2023-04-12 13:56:12,318 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/16cbfdd1cbcb48a08121637f26826480 2023-04-12 13:56:12,333 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/8df74b873c97491cab54f68c7ffc2a76 2023-04-12 13:56:12,336 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c663ddbb2ad44c58920bfc0b70aa20c9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c663ddbb2ad44c58920bfc0b70aa20c9 2023-04-12 13:56:12,340 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c663ddbb2ad44c58920bfc0b70aa20c9, entries=100, sequenceid=11, filesize=18.4 K 2023-04-12 13:56:12,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681307772321, max timestamp=1681307772337Num uniques 1062; 2023-04-12 13:56:12,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-12 13:56:12,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681307772321, max timestamp=1681307772337Num uniques 1062; 2023-04-12 13:56:12,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681307772321, max timestamp=1681307772337Num uniques 1062; 2023-04-12 13:56:12,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=6870, suffix heap size=138416,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=150 2023-04-12 13:56:12,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-12 13:56:12,342 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/16cbfdd1cbcb48a08121637f26826480 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/16cbfdd1cbcb48a08121637f26826480 2023-04-12 13:56:12,345 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/16cbfdd1cbcb48a08121637f26826480, entries=100, sequenceid=11, filesize=9.3 K 2023-04-12 13:56:12,346 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/8df74b873c97491cab54f68c7ffc2a76 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8df74b873c97491cab54f68c7ffc2a76 2023-04-12 13:56:12,350 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8df74b873c97491cab54f68c7ffc2a76, entries=100, sequenceid=11, filesize=9.3 K 2023-04-12 13:56:12,351 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=102.34 KB/104798 for e35088e2f9a6947abbfc447493169cd6 in 482ms, sequenceid=11, compaction requested=false 2023-04-12 13:56:12,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-12 13:56:12,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:12,351 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:12,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:12,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:12,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=122.47 KB heapSize=344.03 KB 2023-04-12 13:56:12,352 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:12,352 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:12,352 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:12,352 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:12,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:12,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,354 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,354 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:12,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296194df[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:12,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:12,356 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,358 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,358 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:12,358 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:12,358 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:12,358 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,358 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:12,359 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,359 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,359 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,360 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:12,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:12,362 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:12,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:12,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:12,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:12,363 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:12,365 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,365 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,366 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,372 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307832368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,374 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307832371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307832372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307832373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307832373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,377 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a165123ef3d34f56a6ceef9de8cf98a5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a165123ef3d34f56a6ceef9de8cf98a5 2023-04-12 13:56:12,378 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,378 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,378 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:12,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:12,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:12,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,378 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,379 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:12,379 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:12,378 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,379 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,379 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,379 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,379 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,379 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,418 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/18442e2b377a4b659060536064fe486f 2023-04-12 13:56:12,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/8b556f24dc3747d2b836ca8c41306729 2023-04-12 13:56:12,451 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.49 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/615721d738bf483c9b6a88dc651e8612 2023-04-12 13:56:12,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/18442e2b377a4b659060536064fe486f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/18442e2b377a4b659060536064fe486f 2023-04-12 13:56:12,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/18442e2b377a4b659060536064fe486f, entries=150, sequenceid=337, filesize=25.4 K 2023-04-12 13:56:12,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/8b556f24dc3747d2b836ca8c41306729 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8b556f24dc3747d2b836ca8c41306729 2023-04-12 13:56:12,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8b556f24dc3747d2b836ca8c41306729, entries=150, sequenceid=337, filesize=11.9 K 2023-04-12 13:56:12,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/615721d738bf483c9b6a88dc651e8612 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/615721d738bf483c9b6a88dc651e8612 2023-04-12 13:56:12,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/615721d738bf483c9b6a88dc651e8612, entries=150, sequenceid=337, filesize=11.9 K 2023-04-12 13:56:12,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~122.47 KB/125408, heapSize ~343.98 KB/352240, currentSize=67.09 KB/68700 for e35088e2f9a6947abbfc447493169cd6 in 118ms, sequenceid=337, compaction requested=false 2023-04-12 13:56:12,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:12,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-12 13:56:12,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:12,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:12,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:12,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:12,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:12,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:12,479 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:12,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:12,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:12,485 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:12,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:12,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:12,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,488 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,488 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:12,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:12,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:12,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:12,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:12,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:12,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:12,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307832487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:12,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:12,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307832488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,491 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:12,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:12,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307832489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,493 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304124600869cf7434b82bf8c2a8b143359af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124600869cf7434b82bf8c2a8b143359af 2023-04-12 13:56:12,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307832491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307832491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,497 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,497 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,498 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:12,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a6b677[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:12,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:12,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,502 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:12,502 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,502 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,504 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:12,507 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:12,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:12,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:12,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:12,507 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:12,510 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,510 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,510 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,511 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,512 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,514 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,514 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,514 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,514 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:12,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:12,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:12,515 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:12,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,515 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:12,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,515 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307832592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307832593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307832593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307832595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307832595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,615 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:12,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:12,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:12,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:12,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:12,618 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:12,618 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:12,619 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,619 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,621 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,621 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,622 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,622 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:12,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:12,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:12,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:12,623 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,623 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:12,623 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:12,623 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:12,623 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:12,625 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,625 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a21aae[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:12,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:12,627 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,628 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:12,629 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,629 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:12,630 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,630 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,631 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:12,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:12,633 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:12,633 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:12,633 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:12,633 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:12,633 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:12,636 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,636 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,636 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,636 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,638 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,640 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:12,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:12,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:12,640 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,640 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,640 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,640 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,640 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,641 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,641 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,641 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,641 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,641 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:12,641 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:12,741 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:12,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:12,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:12,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:12,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:12,745 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:12,745 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,745 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:12,745 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:12,745 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,745 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,747 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,747 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:12,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:12,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:12,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,748 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:12,749 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,749 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:12,749 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:12,749 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:12,750 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,750 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,751 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:12,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d229647[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:12,753 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:12,753 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,755 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:12,755 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,755 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:12,757 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:12,760 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:12,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:12,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:12,760 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:12,760 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:12,761 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,762 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,762 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,764 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,766 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,767 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,767 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:12,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:12,767 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:12,767 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:12,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307832797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307832797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307832797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307832798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:12,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307832798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,867 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:12,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:12,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:12,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:12,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:12,871 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:12,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:12,871 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:12,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,874 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,874 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:12,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:12,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:12,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:12,876 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:12,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:12,878 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68402450[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:12,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:12,880 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,881 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,882 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:12,882 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:12,882 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:12,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:12,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:12,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:12,889 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:12,889 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:12,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:12,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:12,890 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:12,891 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:12,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:12,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,892 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:12,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,894 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:12,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:12,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:12,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:12,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:12,898 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:12,898 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:12,898 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:12,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:12,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:12,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:12,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:12,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:12,900 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=351, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c3d03aee6bfb4aee877e0c42742a52fa 2023-04-12 13:56:12,910 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/826c8d5fc2a44c79becabc006b3951cd 2023-04-12 13:56:12,998 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:12,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:13,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:13,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:13,002 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:13,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:13,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:13,003 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,003 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:13,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:13,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:13,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:13,005 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:13,005 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:13,006 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:13,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,009 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,009 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,009 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:13,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb63221[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:13,012 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:13,012 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,013 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,013 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:13,014 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:13,014 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:13,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:13,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:13,019 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:13,019 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:13,019 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:13,019 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:13,019 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:13,021 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,021 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,021 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,021 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:13,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:13,026 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:13,027 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:13,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,027 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:13,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,027 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307833098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307833100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307833103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307833103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307833104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,127 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:13,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:13,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:13,131 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:13,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:13,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:13,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,131 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,132 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,133 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,133 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:13,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:13,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:13,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:13,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:13,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:13,134 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:13,134 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,135 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:13,135 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:13,135 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:13,137 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:13,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf63e93[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:13,139 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:13,139 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,140 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:13,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:13,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:13,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,141 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:13,143 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:13,143 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,143 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,143 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:13,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:13,146 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:13,146 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:13,146 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:13,146 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:13,146 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:13,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,148 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,150 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:13,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:13,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,152 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,153 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,153 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,153 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,153 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:13,153 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:13,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:13,253 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:13,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:13,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:13,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:13,261 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:13,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:13,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:13,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,263 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,263 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:13,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:13,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:13,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:13,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:13,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:13,272 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:13,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:13,273 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:13,275 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,275 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,276 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:13,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bf17e4[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:13,277 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:13,278 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,279 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:13,279 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:13,279 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,279 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:13,281 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,281 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:13,284 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:13,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:13,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:13,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:13,284 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:13,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,289 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,292 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:13,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:13,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,295 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,295 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,295 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,295 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:13,295 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:13,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/8249707d921d464a92d8ac2331152e84 2023-04-12 13:56:13,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c3d03aee6bfb4aee877e0c42742a52fa as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c3d03aee6bfb4aee877e0c42742a52fa 2023-04-12 13:56:13,328 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c3d03aee6bfb4aee877e0c42742a52fa, entries=200, sequenceid=351, filesize=32.3 K 2023-04-12 13:56:13,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/826c8d5fc2a44c79becabc006b3951cd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/826c8d5fc2a44c79becabc006b3951cd 2023-04-12 13:56:13,332 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/826c8d5fc2a44c79becabc006b3951cd, entries=150, sequenceid=351, filesize=11.9 K 2023-04-12 13:56:13,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/8249707d921d464a92d8ac2331152e84 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8249707d921d464a92d8ac2331152e84 2023-04-12 13:56:13,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8249707d921d464a92d8ac2331152e84, entries=150, sequenceid=351, filesize=11.9 K 2023-04-12 13:56:13,336 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for e35088e2f9a6947abbfc447493169cd6 in 860ms, sequenceid=351, compaction requested=true 2023-04-12 13:56:13,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:13,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:13,337 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:13,337 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:13,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:13,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:13,338 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77930 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:13,338 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction (all files) 2023-04-12 13:56:13,338 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,338 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:13,338 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c663ddbb2ad44c58920bfc0b70aa20c9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/18442e2b377a4b659060536064fe486f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c3d03aee6bfb4aee877e0c42742a52fa] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=76.1 K 2023-04-12 13:56:13,338 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction (all files) 2023-04-12 13:56:13,338 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,338 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/16cbfdd1cbcb48a08121637f26826480, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8b556f24dc3747d2b836ca8c41306729, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/826c8d5fc2a44c79becabc006b3951cd] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=33.1 K 2023-04-12 13:56:13,338 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c663ddbb2ad44c58920bfc0b70aa20c9, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681307771861 2023-04-12 13:56:13,339 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 16cbfdd1cbcb48a08121637f26826480, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681307771861 2023-04-12 13:56:13,339 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 18442e2b377a4b659060536064fe486f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681307772349 2023-04-12 13:56:13,339 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8b556f24dc3747d2b836ca8c41306729, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681307772349 2023-04-12 13:56:13,339 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c3d03aee6bfb4aee877e0c42742a52fa, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681307772370 2023-04-12 13:56:13,339 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 826c8d5fc2a44c79becabc006b3951cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681307772370 2023-04-12 13:56:13,347 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#519 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:13,351 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#518 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:13,358 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/40dd8774a12048cc9cf61ab7735a29e0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/40dd8774a12048cc9cf61ab7735a29e0 2023-04-12 13:56:13,362 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into 40dd8774a12048cc9cf61ab7735a29e0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:13,362 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:13,363 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=13, startTime=1681307773337; duration=0sec 2023-04-12 13:56:13,363 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:13,363 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:13,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:13,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction (all files) 2023-04-12 13:56:13,364 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,364 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8df74b873c97491cab54f68c7ffc2a76, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/615721d738bf483c9b6a88dc651e8612, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8249707d921d464a92d8ac2331152e84] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=33.1 K 2023-04-12 13:56:13,364 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8df74b873c97491cab54f68c7ffc2a76, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681307771861 2023-04-12 13:56:13,365 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 615721d738bf483c9b6a88dc651e8612, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681307772349 2023-04-12 13:56:13,365 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8249707d921d464a92d8ac2331152e84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681307772370 2023-04-12 13:56:13,369 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#520 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:13,381 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/7618be663d7d4cb7be93db695ca96654 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7618be663d7d4cb7be93db695ca96654 2023-04-12 13:56:13,385 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into 7618be663d7d4cb7be93db695ca96654(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:13,385 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:13,385 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=13, startTime=1681307773337; duration=0sec 2023-04-12 13:56:13,386 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:13,395 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:13,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:13,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:13,398 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,399 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,401 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,401 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,402 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:13,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:13,403 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:13,403 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,403 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:13,403 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-12 13:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:13,404 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:13,418 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412864719f00fd34aacbc035ff1b1910070 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412864719f00fd34aacbc035ff1b1910070 2023-04-12 13:56:13,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:13,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:13,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307833614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307833615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307833615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307833616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307833616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307833719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307833719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44798 deadline: 1681307833719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307833719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307833719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,786 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/267b888950994d9d85d865db979f7189 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/267b888950994d9d85d865db979f7189 2023-04-12 13:56:13,790 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into 267b888950994d9d85d865db979f7189(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:13,790 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:13,790 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=13, startTime=1681307773337; duration=0sec 2023-04-12 13:56:13,790 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:13,829 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c138289111db43b0aeec78524a3f7d75 2023-04-12 13:56:13,839 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/51f1ac7fc1ba4b77a7c23aba92f43802 2023-04-12 13:56:13,849 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/f6bab67dc6cf4353ad896b961f063574 2023-04-12 13:56:13,854 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c138289111db43b0aeec78524a3f7d75 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c138289111db43b0aeec78524a3f7d75 2023-04-12 13:56:13,858 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c138289111db43b0aeec78524a3f7d75, entries=150, sequenceid=372, filesize=25.4 K 2023-04-12 13:56:13,858 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/51f1ac7fc1ba4b77a7c23aba92f43802 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/51f1ac7fc1ba4b77a7c23aba92f43802 2023-04-12 13:56:13,861 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/51f1ac7fc1ba4b77a7c23aba92f43802, entries=150, sequenceid=372, filesize=11.9 K 2023-04-12 13:56:13,862 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/f6bab67dc6cf4353ad896b961f063574 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f6bab67dc6cf4353ad896b961f063574 2023-04-12 13:56:13,873 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f6bab67dc6cf4353ad896b961f063574, entries=150, sequenceid=372, filesize=11.9 K 2023-04-12 13:56:13,874 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for e35088e2f9a6947abbfc447493169cd6 in 471ms, sequenceid=372, compaction requested=false 2023-04-12 13:56:13,874 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:13,874 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:13,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:13,876 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,876 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,877 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,877 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:13,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb4dda7[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:13,878 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:13,878 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,885 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:13,885 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:13,885 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,885 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:13,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:13,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,887 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:13,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:13,890 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:13,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:13,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:13,890 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:13,890 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:13,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:13,892 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:13,893 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,895 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:13,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,897 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:13,897 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:13,897 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,897 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:13,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:13,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:13,897 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:13,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:13,898 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:13,897 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:13,898 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:13,898 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:13,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:13,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:13,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:13,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:56:13,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:13,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:13,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:13,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:13,924 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:13,924 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:13,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307833935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307833935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307833936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307833936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:13,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307833937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:13,947 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412157293e289154e4f99ba4df615cd465a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412157293e289154e4f99ba4df615cd465a 2023-04-12 13:56:13,952 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/87859ae4a55c47dc8f0a31c741fc1f32 2023-04-12 13:56:13,966 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/6546580a9ef54255b29c73867624e4e0 2023-04-12 13:56:13,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/02efa948545d4d5ab916f6f7c7f325a1 2023-04-12 13:56:13,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/87859ae4a55c47dc8f0a31c741fc1f32 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/87859ae4a55c47dc8f0a31c741fc1f32 2023-04-12 13:56:13,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/87859ae4a55c47dc8f0a31c741fc1f32, entries=150, sequenceid=390, filesize=25.4 K 2023-04-12 13:56:13,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/6546580a9ef54255b29c73867624e4e0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6546580a9ef54255b29c73867624e4e0 2023-04-12 13:56:13,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6546580a9ef54255b29c73867624e4e0, entries=150, sequenceid=390, filesize=11.9 K 2023-04-12 13:56:13,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/02efa948545d4d5ab916f6f7c7f325a1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/02efa948545d4d5ab916f6f7c7f325a1 2023-04-12 13:56:13,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/02efa948545d4d5ab916f6f7c7f325a1, entries=150, sequenceid=390, filesize=11.9 K 2023-04-12 13:56:13,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for e35088e2f9a6947abbfc447493169cd6 in 68ms, sequenceid=390, compaction requested=true 2023-04-12 13:56:13,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:13,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:13,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:13,992 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:13,992 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:13,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:13,993 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78241 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:13,993 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:13,993 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction (all files) 2023-04-12 13:56:13,993 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction (all files) 2023-04-12 13:56:13,993 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,993 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:13,993 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/267b888950994d9d85d865db979f7189, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c138289111db43b0aeec78524a3f7d75, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/87859ae4a55c47dc8f0a31c741fc1f32] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=76.4 K 2023-04-12 13:56:13,993 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/40dd8774a12048cc9cf61ab7735a29e0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/51f1ac7fc1ba4b77a7c23aba92f43802, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6546580a9ef54255b29c73867624e4e0] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=35.8 K 2023-04-12 13:56:13,993 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 267b888950994d9d85d865db979f7189, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681307772370 2023-04-12 13:56:13,994 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 40dd8774a12048cc9cf61ab7735a29e0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681307772370 2023-04-12 13:56:13,994 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c138289111db43b0aeec78524a3f7d75, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681307772488 2023-04-12 13:56:13,994 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 51f1ac7fc1ba4b77a7c23aba92f43802, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681307772488 2023-04-12 13:56:13,994 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 87859ae4a55c47dc8f0a31c741fc1f32, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681307773922 2023-04-12 13:56:13,994 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6546580a9ef54255b29c73867624e4e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681307773922 2023-04-12 13:56:13,998 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:14,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:14,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:14,012 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:14,012 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,012 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:14,012 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:14,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,013 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,016 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,016 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,016 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,016 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:14,025 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:14,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:14,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,036 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:14,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:56:14,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:14,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:14,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:14,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:14,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:14,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:14,043 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,043 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:14,043 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:14,043 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:14,043 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:14,045 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#528 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:14,045 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,045 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,046 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,047 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:14,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a9c12a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:14,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:14,048 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,049 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:14,050 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:14,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,050 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:14,051 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:14,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:14,055 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:14,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:14,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:14,055 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:14,055 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:14,059 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/20a85e956d54481bb885ef20f6900091 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/20a85e956d54481bb885ef20f6900091 2023-04-12 13:56:14,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307834053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307834053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307834054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307834056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307834058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,063 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into 20a85e956d54481bb885ef20f6900091(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:14,063 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:14,063 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=13, startTime=1681307773992; duration=0sec 2023-04-12 13:56:14,064 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:14,064 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:14,064 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,064 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,064 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,065 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,067 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,067 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction (all files) 2023-04-12 13:56:14,067 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,067 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7618be663d7d4cb7be93db695ca96654, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f6bab67dc6cf4353ad896b961f063574, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/02efa948545d4d5ab916f6f7c7f325a1] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=35.8 K 2023-04-12 13:56:14,067 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,069 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7618be663d7d4cb7be93db695ca96654, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681307772370 2023-04-12 13:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,069 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f6bab67dc6cf4353ad896b961f063574, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681307772488 2023-04-12 13:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,070 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 02efa948545d4d5ab916f6f7c7f325a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681307773922 2023-04-12 13:56:14,070 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,071 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,071 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:14,071 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:14,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:14,073 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412fbb29e3dd636483dabe9ec25cf8f6825 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fbb29e3dd636483dabe9ec25cf8f6825 2023-04-12 13:56:14,082 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#530 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:14,082 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=407, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/3880ddc958a143c099e6f3e935b26d77 2023-04-12 13:56:14,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/8054379535da4b3384b3e5bfa2de090d 2023-04-12 13:56:14,095 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/8445e94adccf4d9aae8a40efeac2bf68 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8445e94adccf4d9aae8a40efeac2bf68 2023-04-12 13:56:14,102 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into 8445e94adccf4d9aae8a40efeac2bf68(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:14,102 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:14,102 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=13, startTime=1681307773992; duration=0sec 2023-04-12 13:56:14,102 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307834161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307834161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307834161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307834161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307834162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,168 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 13:56:14,171 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:14,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:14,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:14,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:14,180 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:14,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:14,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:14,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,180 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,182 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:14,188 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:14,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:14,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,194 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:14,196 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,196 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:14,196 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:14,196 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:14,196 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:14,198 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,199 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:14,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba663ca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:14,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:14,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,203 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:14,203 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:14,203 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:14,203 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,203 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:14,205 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,205 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,205 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,205 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:14,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:14,208 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:14,208 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:14,208 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:14,208 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:14,208 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:14,211 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,211 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,211 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,212 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,213 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,219 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,219 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:14,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,219 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,219 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:14,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,320 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:14,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:14,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:14,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:14,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:14,330 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:14,330 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:14,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:14,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,339 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,339 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,340 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,340 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:14,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:14,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,341 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:14,348 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,348 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:14,348 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:14,348 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:14,348 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:14,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,350 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,350 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,350 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:14,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263b536d[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:14,352 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:14,352 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,354 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,354 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:14,354 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,354 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:14,354 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:14,354 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:14,356 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,356 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:14,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:14,358 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:14,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:14,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:14,359 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:14,359 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:14,361 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,361 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,361 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,366 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,366 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,367 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307834364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:14,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:14,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,366 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:14,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307834365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,368 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:14,368 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:14,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307834366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307834366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307834367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,468 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:14,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:14,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:14,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:14,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:14,472 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:14,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:14,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:14,473 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,473 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,475 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,475 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,475 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,475 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:14,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:14,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:14,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:14,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:14,477 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:14,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:14,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:14,477 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,478 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:14,478 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:14,478 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,478 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:14,478 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:14,481 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,481 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,482 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:14,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:14,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc8fbe8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:14,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:14,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,485 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:14,485 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:14,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,485 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:14,486 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:14,486 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,487 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:14,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:14,490 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:14,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:14,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:14,490 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:14,490 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:14,494 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,494 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,494 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:14,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,499 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/50f1a52c50684617b1f27082fe8b1919 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/50f1a52c50684617b1f27082fe8b1919 2023-04-12 13:56:14,500 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,501 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,504 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into 50f1a52c50684617b1f27082fe8b1919(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:14,504 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:14,504 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=13, startTime=1681307773992; duration=0sec 2023-04-12 13:56:14,504 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:14,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,505 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:14,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:14,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:14,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,506 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:14,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:14,506 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:14,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:14,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,506 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:14,506 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:14,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/ca2815cbfb0742ceaa3c1090c8208951 2023-04-12 13:56:14,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/3880ddc958a143c099e6f3e935b26d77 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3880ddc958a143c099e6f3e935b26d77 2023-04-12 13:56:14,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3880ddc958a143c099e6f3e935b26d77, entries=200, sequenceid=407, filesize=32.3 K 2023-04-12 13:56:14,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/8054379535da4b3384b3e5bfa2de090d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8054379535da4b3384b3e5bfa2de090d 2023-04-12 13:56:14,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8054379535da4b3384b3e5bfa2de090d, entries=150, sequenceid=407, filesize=11.9 K 2023-04-12 13:56:14,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/ca2815cbfb0742ceaa3c1090c8208951 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/ca2815cbfb0742ceaa3c1090c8208951 2023-04-12 13:56:14,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/ca2815cbfb0742ceaa3c1090c8208951, entries=150, sequenceid=407, filesize=11.9 K 2023-04-12 13:56:14,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e35088e2f9a6947abbfc447493169cd6 in 489ms, sequenceid=407, compaction requested=false 2023-04-12 13:56:14,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:14,607 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:14,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:14,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:14,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:14,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:14,610 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:14,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:14,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:14,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,612 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:14,612 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:14,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,613 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,613 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:14,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:14,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:14,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:14,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:14,614 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:14,614 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:14,614 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:14,615 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-12 13:56:14,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:14,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:14,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:14,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:14,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:14,615 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:14,628 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412832c1ec197f442a6b504ca9ff82f43b7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412832c1ec197f442a6b504ca9ff82f43b7 2023-04-12 13:56:14,634 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/993babcef0014deab01a0ece083a1615 2023-04-12 13:56:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:14,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:14,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307834680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307834681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307834681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307834681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307834684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307834785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307834785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307834785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307834785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307834788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307834988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307834989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307834989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307834989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:14,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:14,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307834990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,046 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/54a9288417ec4669936896b0a37f5d8d 2023-04-12 13:56:15,056 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/04a70e82f7394d3db0c506eff032a9af 2023-04-12 13:56:15,061 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/993babcef0014deab01a0ece083a1615 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/993babcef0014deab01a0ece083a1615 2023-04-12 13:56:15,064 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/993babcef0014deab01a0ece083a1615, entries=150, sequenceid=425, filesize=25.4 K 2023-04-12 13:56:15,065 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/54a9288417ec4669936896b0a37f5d8d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/54a9288417ec4669936896b0a37f5d8d 2023-04-12 13:56:15,068 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/54a9288417ec4669936896b0a37f5d8d, entries=150, sequenceid=425, filesize=11.9 K 2023-04-12 13:56:15,069 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/04a70e82f7394d3db0c506eff032a9af as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/04a70e82f7394d3db0c506eff032a9af 2023-04-12 13:56:15,072 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/04a70e82f7394d3db0c506eff032a9af, entries=150, sequenceid=425, filesize=11.9 K 2023-04-12 13:56:15,073 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e35088e2f9a6947abbfc447493169cd6 in 458ms, sequenceid=425, compaction requested=true 2023-04-12 13:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:15,073 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:15,073 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:15,073 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,073 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:15,073 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:15,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,076 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,076 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,076 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:15,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7410cbca[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:15,078 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:15,078 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,079 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,079 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:15,079 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,079 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:15,079 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:15,079 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:15,081 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,081 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:15,081 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,081 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:15,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:15,084 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:15,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:15,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:15,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:15,085 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:15,087 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,087 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,087 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:15,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,087 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,089 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,093 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:15,093 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:15,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:15,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:15,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,094 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,094 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:15,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:15,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:15,095 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:15,095 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:15,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:15,195 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:15,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:15,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:15,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:15,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:15,198 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:15,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:15,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:15,198 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:15,199 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,199 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,200 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:15,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:15,200 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,201 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:15,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:15,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:15,202 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:15,202 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:15,202 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:15,202 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:15,202 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:15,202 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:15,202 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:15,202 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-12 13:56:15,202 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:15,202 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,203 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:15,203 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,203 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:15,203 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:15,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:15,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307835300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307835303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307835304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307835305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307835305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307835406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307835407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307835408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44766 deadline: 1681307835410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307835410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44752 deadline: 1681307835609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307835611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307835612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307835612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:15,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307835612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,621 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304125f4aef7b4c7d4a44b40eca159cd0cafb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125f4aef7b4c7d4a44b40eca159cd0cafb 2023-04-12 13:56:15,630 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=442, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/d4e02164041e4dfa8bb84f38ca721219 2023-04-12 13:56:15,644 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/a2a8dffba135422bbbfc07dca64d66cf 2023-04-12 13:56:15,658 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/752e2b88a5a348e9a608ffc19ee4a040 2023-04-12 13:56:15,663 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/d4e02164041e4dfa8bb84f38ca721219 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d4e02164041e4dfa8bb84f38ca721219 2023-04-12 13:56:15,668 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d4e02164041e4dfa8bb84f38ca721219, entries=150, sequenceid=442, filesize=25.4 K 2023-04-12 13:56:15,668 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/a2a8dffba135422bbbfc07dca64d66cf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/a2a8dffba135422bbbfc07dca64d66cf 2023-04-12 13:56:15,672 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/a2a8dffba135422bbbfc07dca64d66cf, entries=150, sequenceid=442, filesize=11.9 K 2023-04-12 13:56:15,672 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/752e2b88a5a348e9a608ffc19ee4a040 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/752e2b88a5a348e9a608ffc19ee4a040 2023-04-12 13:56:15,679 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/752e2b88a5a348e9a608ffc19ee4a040, entries=150, sequenceid=442, filesize=11.9 K 2023-04-12 13:56:15,679 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for e35088e2f9a6947abbfc447493169cd6 in 477ms, sequenceid=442, compaction requested=true 2023-04-12 13:56:15,681 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:15,681 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:15,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:15,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:15,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,681 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:15,683 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,683 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,684 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:15,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca04eb1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:15,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:15,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,687 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:15,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:15,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:15,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,688 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:15,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:15,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:15,692 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:15,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:15,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:15,692 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:15,692 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:15,693 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:15,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,694 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,695 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:15,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:15,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:15,698 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:15,699 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,699 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,699 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:15,699 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:15,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:15,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:15,699 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:15,699 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:15,799 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:15,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:15,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:15,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:15,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:15,803 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:15,803 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:15,803 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:15,803 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:15,803 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,804 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,805 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:15,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:15,806 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,806 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:15,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:15,807 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:15,807 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:15,808 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-12 13:56:15,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:15,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:15,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:15,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,828 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412e5a4c8c2b82844efb88e6221b939c2d6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e5a4c8c2b82844efb88e6221b939c2d6 2023-04-12 13:56:15,837 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=458, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/bf395584a4cd472a9aaf347b4b1418ed 2023-04-12 13:56:15,851 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/762405ea73214168b4e453cc1d1f7dd1 2023-04-12 13:56:15,862 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/d3467cb1795842968e302127ee92f5b9 2023-04-12 13:56:15,866 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/bf395584a4cd472a9aaf347b4b1418ed as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/bf395584a4cd472a9aaf347b4b1418ed 2023-04-12 13:56:15,870 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/bf395584a4cd472a9aaf347b4b1418ed, entries=150, sequenceid=458, filesize=25.4 K 2023-04-12 13:56:15,871 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/762405ea73214168b4e453cc1d1f7dd1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/762405ea73214168b4e453cc1d1f7dd1 2023-04-12 13:56:15,880 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/762405ea73214168b4e453cc1d1f7dd1, entries=150, sequenceid=458, filesize=11.9 K 2023-04-12 13:56:15,881 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/d3467cb1795842968e302127ee92f5b9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d3467cb1795842968e302127ee92f5b9 2023-04-12 13:56:15,897 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d3467cb1795842968e302127ee92f5b9, entries=150, sequenceid=458, filesize=11.9 K 2023-04-12 13:56:15,898 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for e35088e2f9a6947abbfc447493169cd6 in 91ms, sequenceid=458, compaction requested=true 2023-04-12 13:56:15,898 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:15,898 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:15,898 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:15,899 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:15,900 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,900 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,901 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:15,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda73[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:15,903 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:15,903 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,904 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:15,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:15,904 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:15,904 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,904 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:15,905 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:15,905 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:15,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:15,908 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:15,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:15,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:15,908 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:15,909 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:15,911 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,911 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:15,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,911 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:15,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:15,912 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,914 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:15,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:15,918 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:15,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:15,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:15,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:15,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:15,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:15,919 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:15,919 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:15,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:15,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:15,949 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:15,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:15,977 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041280b4b98b14194aa3ba9a5a5694747321 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041280b4b98b14194aa3ba9a5a5694747321 2023-04-12 13:56:16,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307775972, max timestamp=1681307776002Num uniques 859; 2023-04-12 13:56:16,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307775972, max timestamp=1681307776002Num uniques 859; 2023-04-12 13:56:16,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:16,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:16,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307775972, max timestamp=1681307776002Num uniques 859; 2023-04-12 13:56:16,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:16,019 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:16,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:16,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:16,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:16,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:16,022 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:16,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:16,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:16,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,022 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,024 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,024 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:16,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:16,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:16,026 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,026 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:16,026 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:16,026 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:16,028 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,028 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,029 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:16,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307836022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b20762f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:16,030 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:16,030 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307836026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307836027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,033 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:16,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:16,033 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:16,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,033 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:16,034 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,034 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:16,034 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,034 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:16,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307836028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307836028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:16,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:16,037 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:16,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:16,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:16,037 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:16,037 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:16,038 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,038 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,038 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,039 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,040 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,044 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:16,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,044 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,044 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:16,045 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:16,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307836129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307836131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307836132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307836135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307836136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,145 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:16,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:16,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:16,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:16,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:16,148 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:16,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:16,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:16,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,149 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,151 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:16,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:16,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:16,153 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,153 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:16,153 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:16,153 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:16,153 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:16,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:16,154 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:16,158 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,158 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399caa1a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:16,159 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:16,160 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,161 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:16,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,162 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:16,168 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:16,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:16,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:16,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:16,168 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:16,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,171 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,176 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,176 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:16,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:16,176 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:16,176 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:16,276 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:16,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:16,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:16,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:16,280 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:16,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:16,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:16,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,280 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,281 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,282 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:16,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:16,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:16,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:16,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:16,283 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:16,283 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,283 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:16,283 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:16,284 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:16,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:16,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4737ae5e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:16,289 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:16,289 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,290 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,290 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,290 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:16,291 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,293 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:16,296 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:16,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:16,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:16,296 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:16,296 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:16,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,297 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,298 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,300 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:16,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:16,303 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,303 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,303 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:16,304 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:16,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307836336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307836336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307836337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307836341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307836341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,384 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=469, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/5dd2dc5556e547e5a75f6cb551a87369 2023-04-12 13:56:16,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ac756d4c75f9430c908362746683a870 2023-04-12 13:56:16,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/d50cfced1489458bb5bc3ad5a6a7f2fb 2023-04-12 13:56:16,404 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:16,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:16,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/5dd2dc5556e547e5a75f6cb551a87369 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5dd2dc5556e547e5a75f6cb551a87369 2023-04-12 13:56:16,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:16,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:16,407 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:16,407 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,407 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:16,407 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:16,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,408 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,409 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,409 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,410 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:16,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:16,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:16,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5dd2dc5556e547e5a75f6cb551a87369, entries=250, sequenceid=469, filesize=39.2 K 2023-04-12 13:56:16,411 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,411 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:16,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ac756d4c75f9430c908362746683a870 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac756d4c75f9430c908362746683a870 2023-04-12 13:56:16,413 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,414 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,414 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:16,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217e5b89[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:16,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:16,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac756d4c75f9430c908362746683a870, entries=150, sequenceid=469, filesize=11.9 K 2023-04-12 13:56:16,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:16,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/d50cfced1489458bb5bc3ad5a6a7f2fb as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d50cfced1489458bb5bc3ad5a6a7f2fb 2023-04-12 13:56:16,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,417 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:16,419 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:16,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d50cfced1489458bb5bc3ad5a6a7f2fb, entries=150, sequenceid=469, filesize=11.9 K 2023-04-12 13:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for e35088e2f9a6947abbfc447493169cd6 in 472ms, sequenceid=469, compaction requested=true 2023-04-12 13:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:16,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:16,421 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:56:16,421 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:56:16,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:16,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:16,422 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:16,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:16,423 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:16,423 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:16,423 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:16,423 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73322 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:56:16,423 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 177606 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:56:16,423 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction (all files) 2023-04-12 13:56:16,423 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction (all files) 2023-04-12 13:56:16,423 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,423 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,423 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/20a85e956d54481bb885ef20f6900091, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8054379535da4b3384b3e5bfa2de090d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/54a9288417ec4669936896b0a37f5d8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/a2a8dffba135422bbbfc07dca64d66cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/762405ea73214168b4e453cc1d1f7dd1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac756d4c75f9430c908362746683a870] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=71.6 K 2023-04-12 13:56:16,423 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/50f1a52c50684617b1f27082fe8b1919, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3880ddc958a143c099e6f3e935b26d77, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/993babcef0014deab01a0ece083a1615, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d4e02164041e4dfa8bb84f38ca721219, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/bf395584a4cd472a9aaf347b4b1418ed, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5dd2dc5556e547e5a75f6cb551a87369] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=173.4 K 2023-04-12 13:56:16,424 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 20a85e956d54481bb885ef20f6900091, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681307773922 2023-04-12 13:56:16,424 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 50f1a52c50684617b1f27082fe8b1919, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681307773922 2023-04-12 13:56:16,424 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8054379535da4b3384b3e5bfa2de090d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681307773935 2023-04-12 13:56:16,425 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 54a9288417ec4669936896b0a37f5d8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681307774050 2023-04-12 13:56:16,425 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3880ddc958a143c099e6f3e935b26d77, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681307773928 2023-04-12 13:56:16,425 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a2a8dffba135422bbbfc07dca64d66cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681307774680 2023-04-12 13:56:16,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,425 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 993babcef0014deab01a0ece083a1615, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681307774050 2023-04-12 13:56:16,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:16,425 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 762405ea73214168b4e453cc1d1f7dd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1681307775298 2023-04-12 13:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:16,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,426 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d4e02164041e4dfa8bb84f38ca721219, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681307774680 2023-04-12 13:56:16,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,426 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ac756d4c75f9430c908362746683a870, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681307775938 2023-04-12 13:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,427 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting bf395584a4cd472a9aaf347b4b1418ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1681307775298 2023-04-12 13:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:16,427 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5dd2dc5556e547e5a75f6cb551a87369, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681307775937 2023-04-12 13:56:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:16,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:16,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:16,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,433 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:16,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:16,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,433 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:16,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:16,441 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#545 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:16,442 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#546 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:16,462 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/4c2baa1173934e058670314032cf65c8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/4c2baa1173934e058670314032cf65c8 2023-04-12 13:56:16,467 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into 4c2baa1173934e058670314032cf65c8(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:16,467 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:16,467 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=10, startTime=1681307776421; duration=0sec 2023-04-12 13:56:16,467 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:16,468 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:56:16,470 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73322 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:56:16,470 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction (all files) 2023-04-12 13:56:16,470 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,470 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8445e94adccf4d9aae8a40efeac2bf68, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/ca2815cbfb0742ceaa3c1090c8208951, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/04a70e82f7394d3db0c506eff032a9af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/752e2b88a5a348e9a608ffc19ee4a040, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d3467cb1795842968e302127ee92f5b9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d50cfced1489458bb5bc3ad5a6a7f2fb] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=71.6 K 2023-04-12 13:56:16,470 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 8445e94adccf4d9aae8a40efeac2bf68, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681307773922 2023-04-12 13:56:16,471 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ca2815cbfb0742ceaa3c1090c8208951, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681307773935 2023-04-12 13:56:16,471 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 04a70e82f7394d3db0c506eff032a9af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1681307774050 2023-04-12 13:56:16,472 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 752e2b88a5a348e9a608ffc19ee4a040, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681307774680 2023-04-12 13:56:16,472 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d3467cb1795842968e302127ee92f5b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1681307775298 2023-04-12 13:56:16,472 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/f91c89864b684304a221878d4c7abca6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f91c89864b684304a221878d4c7abca6 2023-04-12 13:56:16,473 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting d50cfced1489458bb5bc3ad5a6a7f2fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681307775938 2023-04-12 13:56:16,476 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into f91c89864b684304a221878d4c7abca6(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:16,476 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:16,476 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=10, startTime=1681307776421; duration=0sec 2023-04-12 13:56:16,476 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:16,482 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#547 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:16,492 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/03f2ed7bacc9435f81a5b6b40e1f0472 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/03f2ed7bacc9435f81a5b6b40e1f0472 2023-04-12 13:56:16,497 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into 03f2ed7bacc9435f81a5b6b40e1f0472(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:16,497 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:16,497 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=10, startTime=1681307776421; duration=0sec 2023-04-12 13:56:16,497 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:16,533 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:16,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:16,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:16,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:16,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:16,537 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:16,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:16,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:16,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,537 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,538 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:16,538 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:16,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,539 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,539 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:16,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:16,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:16,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:16,540 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:16,540 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:16,540 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:16,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:16,541 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:16,541 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:16,541 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:16,542 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:16,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:16,643 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:16,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307836647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307836655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307836670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307836670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307836670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307836773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307836775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307836775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307836775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,967 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412667917b413524752bb822b807ac00258 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412667917b413524752bb822b807ac00258 2023-04-12 13:56:16,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307836975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307836979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307836979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307836979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44784 deadline: 1681307837172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307837279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307837282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307837282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307837283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,373 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=508, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/3ad23180649e4f09a712578086113a65 2023-04-12 13:56:17,782 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ac654c086b094fac8c9f231a1301961f 2023-04-12 13:56:17,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307837782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307837785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307837786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307837790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,792 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/58001a5e2cc8454783ddb9821a7377f7 2023-04-12 13:56:17,796 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/3ad23180649e4f09a712578086113a65 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3ad23180649e4f09a712578086113a65 2023-04-12 13:56:17,800 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3ad23180649e4f09a712578086113a65, entries=150, sequenceid=508, filesize=25.4 K 2023-04-12 13:56:17,800 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ac654c086b094fac8c9f231a1301961f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac654c086b094fac8c9f231a1301961f 2023-04-12 13:56:17,803 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac654c086b094fac8c9f231a1301961f, entries=150, sequenceid=508, filesize=11.9 K 2023-04-12 13:56:17,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/58001a5e2cc8454783ddb9821a7377f7 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/58001a5e2cc8454783ddb9821a7377f7 2023-04-12 13:56:17,808 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/58001a5e2cc8454783ddb9821a7377f7, entries=150, sequenceid=508, filesize=11.9 K 2023-04-12 13:56:17,808 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e35088e2f9a6947abbfc447493169cd6 in 1267ms, sequenceid=508, compaction requested=false 2023-04-12 13:56:17,808 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:17,809 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:17,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:17,812 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:17,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a7c087[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:17,814 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:17,814 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,817 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,817 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:17,817 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:17,817 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:17,817 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,817 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:17,821 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:17,824 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:17,824 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:17,824 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:17,824 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:17,824 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:17,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:17,827 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:17,828 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,829 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,832 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:17,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:17,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:17,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:17,832 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:17,833 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:17,833 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:17,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:17,933 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:17,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:17,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:17,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:17,936 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:17,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:17,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:17,936 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:17,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,937 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:17,938 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,938 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:17,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:17,938 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:17,938 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:17,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:17,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:17,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:17,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:17,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:17,939 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:17,939 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:17,939 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:17,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:17,940 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:17,940 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:17,940 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:17,940 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:17,940 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:17,940 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:17,941 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:17,941 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:17,941 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:17,941 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:18,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:18,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:18,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307778217, max timestamp=1681307778243Num uniques 859; 2023-04-12 13:56:18,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:18,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307778217, max timestamp=1681307778243Num uniques 859; 2023-04-12 13:56:18,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307778217, max timestamp=1681307778243Num uniques 859; 2023-04-12 13:56:18,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:18,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:18,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307838294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,362 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b6c51390494748abb6702fee28eba76f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b6c51390494748abb6702fee28eba76f 2023-04-12 13:56:18,368 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/6c9521f2cf39495b90e3613dc848ac7d 2023-04-12 13:56:18,380 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ed24daacbb4b4a87aef35dab77629521 2023-04-12 13:56:18,387 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/f87dd1b87e5844f78c5871b8148d1574 2023-04-12 13:56:18,392 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/6c9521f2cf39495b90e3613dc848ac7d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6c9521f2cf39495b90e3613dc848ac7d 2023-04-12 13:56:18,396 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6c9521f2cf39495b90e3613dc848ac7d, entries=150, sequenceid=518, filesize=25.4 K 2023-04-12 13:56:18,396 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ed24daacbb4b4a87aef35dab77629521 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ed24daacbb4b4a87aef35dab77629521 2023-04-12 13:56:18,400 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ed24daacbb4b4a87aef35dab77629521, entries=150, sequenceid=518, filesize=11.9 K 2023-04-12 13:56:18,400 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/f87dd1b87e5844f78c5871b8148d1574 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f87dd1b87e5844f78c5871b8148d1574 2023-04-12 13:56:18,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307838398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,405 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f87dd1b87e5844f78c5871b8148d1574, entries=150, sequenceid=518, filesize=11.9 K 2023-04-12 13:56:18,406 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e35088e2f9a6947abbfc447493169cd6 in 466ms, sequenceid=518, compaction requested=true 2023-04-12 13:56:18,406 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:18,406 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:18,409 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35377bd3[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:18,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:18,411 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,414 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,414 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:18,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,415 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:18,416 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:18,416 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:18,418 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:18,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:18,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:18,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:18,418 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:18,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:18,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:18,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,420 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:18,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,421 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:18,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:18,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:18,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:18,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:18,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:18,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:18,425 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:18,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:18,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:18,425 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:18,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:18,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:18,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,525 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:18,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:18,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:18,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:18,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:18,528 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,529 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:18,531 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:18,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:18,531 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:18,532 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,532 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:18,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:18,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:18,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:18,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:18,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:18,533 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:18,533 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:18,533 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:18,533 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-12 13:56:18,533 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:18,533 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:18,533 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:18,534 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:18,534 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:18,534 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:18,549 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412d6c9f70a326148eea782ef3b816ff3c8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d6c9f70a326148eea782ef3b816ff3c8 2023-04-12 13:56:18,554 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=555, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/f0ad97f06f5c4c75b20fd0112a9ebaeb 2023-04-12 13:56:18,563 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/f3f9b7f2385c4ecf8670f156863ba4f8 2023-04-12 13:56:18,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:18,606 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:18,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307838636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307838740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307838792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44732 deadline: 1681307838793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307838797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307838798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:18,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307838945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,974 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/e839b7826218411fb9c3240af0ee4d58 2023-04-12 13:56:18,978 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/f0ad97f06f5c4c75b20fd0112a9ebaeb as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f0ad97f06f5c4c75b20fd0112a9ebaeb 2023-04-12 13:56:18,981 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f0ad97f06f5c4c75b20fd0112a9ebaeb, entries=150, sequenceid=555, filesize=25.4 K 2023-04-12 13:56:18,982 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/f3f9b7f2385c4ecf8670f156863ba4f8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f3f9b7f2385c4ecf8670f156863ba4f8 2023-04-12 13:56:18,986 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f3f9b7f2385c4ecf8670f156863ba4f8, entries=150, sequenceid=555, filesize=11.9 K 2023-04-12 13:56:18,987 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/e839b7826218411fb9c3240af0ee4d58 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e839b7826218411fb9c3240af0ee4d58 2023-04-12 13:56:18,990 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e839b7826218411fb9c3240af0ee4d58, entries=150, sequenceid=555, filesize=11.9 K 2023-04-12 13:56:18,991 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e35088e2f9a6947abbfc447493169cd6 in 458ms, sequenceid=555, compaction requested=true 2023-04-12 13:56:18,991 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:18,991 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:18,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:18,993 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f086675[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:18,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:18,995 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,996 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:18,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:18,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:18,996 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:18,996 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,997 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:18,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:19,001 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:19,001 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:19,001 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:19,001 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:19,001 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:19,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:19,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:19,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:19,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,006 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:19,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:19,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:19,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:19,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:19,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:19,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:19,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:19,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,009 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:19,009 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:19,109 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:19,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:19,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:19,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:19,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:19,112 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:19,112 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:19,112 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:19,112 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:19,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,114 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:19,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:19,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:19,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:19,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:19,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:19,116 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:19,116 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:19,116 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:19,130 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041235b8065a4fb6402887652cb431c9e28b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041235b8065a4fb6402887652cb431c9e28b 2023-04-12 13:56:19,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:19,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:19,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307779281, max timestamp=1681307779306Num uniques 859; 2023-04-12 13:56:19,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307779281, max timestamp=1681307779306Num uniques 859; 2023-04-12 13:56:19,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307779281, max timestamp=1681307779306Num uniques 859; 2023-04-12 13:56:19,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:19,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:19,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:19,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:19,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307839378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307839481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,538 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=565, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c1f25d2e449e436eb2f21edae1ea8a5f 2023-04-12 13:56:19,548 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/147fdbd617be4c10b7f1f9818861be8e 2023-04-12 13:56:19,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:19,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307839685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,960 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/721264f6e735469b9a4f726e85e7ab4d 2023-04-12 13:56:19,964 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/c1f25d2e449e436eb2f21edae1ea8a5f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c1f25d2e449e436eb2f21edae1ea8a5f 2023-04-12 13:56:19,967 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c1f25d2e449e436eb2f21edae1ea8a5f, entries=150, sequenceid=565, filesize=25.4 K 2023-04-12 13:56:19,968 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/147fdbd617be4c10b7f1f9818861be8e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/147fdbd617be4c10b7f1f9818861be8e 2023-04-12 13:56:19,972 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/147fdbd617be4c10b7f1f9818861be8e, entries=150, sequenceid=565, filesize=11.9 K 2023-04-12 13:56:19,972 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/721264f6e735469b9a4f726e85e7ab4d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/721264f6e735469b9a4f726e85e7ab4d 2023-04-12 13:56:19,975 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/721264f6e735469b9a4f726e85e7ab4d, entries=150, sequenceid=565, filesize=11.9 K 2023-04-12 13:56:19,976 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e35088e2f9a6947abbfc447493169cd6 in 860ms, sequenceid=565, compaction requested=true 2023-04-12 13:56:19,976 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:19,976 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:19,978 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c79de0e[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:19,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:19,980 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,981 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:19,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,981 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:19,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:19,982 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:19,982 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:19,985 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:19,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:19,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:19,985 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:19,985 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:19,990 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,990 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:19,990 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:19,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:19,990 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,992 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:19,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:19,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-12 13:56:19,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:19,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:19,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:19,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:19,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:19,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:19,997 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:19,997 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:19,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:19,998 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:19,998 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:19,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:20,008 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304128e29f512004443fa8a23d6a90cd770e8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128e29f512004443fa8a23d6a90cd770e8 2023-04-12 13:56:20,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307840019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,098 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:20,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:20,102 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:20,102 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,102 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:20,102 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:20,102 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,102 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,104 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,104 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:20,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:20,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:20,107 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,107 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8d96fe[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:20,110 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:20,110 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,111 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,112 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:20,112 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:20,112 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,112 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:20,112 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:20,113 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:20,116 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:20,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:20,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:20,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:20,116 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:20,118 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,118 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,118 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:20,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,119 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,120 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,122 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,122 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,122 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,122 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,122 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,122 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:20,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:20,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:20,123 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,123 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,123 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:20,123 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:20,123 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:20,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,123 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:20,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:20,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307840125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,227 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:20,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:20,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:20,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:20,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:20,230 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:20,230 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,230 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:20,230 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:20,230 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,231 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,232 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,232 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,233 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,233 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:20,238 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,238 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:20,238 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:20,238 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:20,238 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,241 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ea85bf[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:20,243 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:20,243 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,244 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,244 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,244 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:20,244 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:20,246 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:20,249 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:20,249 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:20,249 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:20,249 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:20,249 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:20,251 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,251 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,251 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:20,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:20,252 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,253 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:20,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:20,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:20,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,256 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:20,255 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:20,255 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,256 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:20,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:20,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,256 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307840330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,356 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:20,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:20,360 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:20,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:20,360 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:20,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,362 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,362 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:20,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:20,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:20,364 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,364 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:20,364 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:20,364 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:20,364 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:20,366 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,366 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,366 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,366 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:20,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49dd78b9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:20,367 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:20,367 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,370 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:20,370 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,370 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:20,370 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:20,372 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:20,374 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:20,374 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:20,374 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:20,374 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:20,374 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:20,376 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,376 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,376 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:20,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:20,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:20,376 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,378 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:20,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,383 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,383 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,383 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,383 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,383 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:20,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,383 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:20,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:20,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:20,383 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,384 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,384 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:20,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:20,384 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,384 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:20,384 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:20,413 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=603, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/804d438e79ff492093af103a8650e97b 2023-04-12 13:56:20,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/bf80b172fe4f47d1b5a81ce12438cdc2 2023-04-12 13:56:20,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/c262ced90e404f399d37c29240fb86f2 2023-04-12 13:56:20,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/804d438e79ff492093af103a8650e97b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/804d438e79ff492093af103a8650e97b 2023-04-12 13:56:20,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/804d438e79ff492093af103a8650e97b, entries=200, sequenceid=603, filesize=32.3 K 2023-04-12 13:56:20,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/bf80b172fe4f47d1b5a81ce12438cdc2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/bf80b172fe4f47d1b5a81ce12438cdc2 2023-04-12 13:56:20,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/bf80b172fe4f47d1b5a81ce12438cdc2, entries=150, sequenceid=603, filesize=11.9 K 2023-04-12 13:56:20,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/c262ced90e404f399d37c29240fb86f2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c262ced90e404f399d37c29240fb86f2 2023-04-12 13:56:20,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c262ced90e404f399d37c29240fb86f2, entries=150, sequenceid=603, filesize=11.9 K 2023-04-12 13:56:20,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for e35088e2f9a6947abbfc447493169cd6 in 454ms, sequenceid=603, compaction requested=true 2023-04-12 13:56:20,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:20,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:20,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:20,447 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:56:20,447 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:56:20,447 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:20,449 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163726 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:56:20,449 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction (all files) 2023-04-12 13:56:20,449 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,449 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f91c89864b684304a221878d4c7abca6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3ad23180649e4f09a712578086113a65, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6c9521f2cf39495b90e3613dc848ac7d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f0ad97f06f5c4c75b20fd0112a9ebaeb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c1f25d2e449e436eb2f21edae1ea8a5f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/804d438e79ff492093af103a8650e97b] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=159.9 K 2023-04-12 13:56:20,449 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73526 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:56:20,449 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f91c89864b684304a221878d4c7abca6, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681307775938 2023-04-12 13:56:20,449 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction (all files) 2023-04-12 13:56:20,449 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,450 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/4c2baa1173934e058670314032cf65c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac654c086b094fac8c9f231a1301961f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ed24daacbb4b4a87aef35dab77629521, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f3f9b7f2385c4ecf8670f156863ba4f8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/147fdbd617be4c10b7f1f9818861be8e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/bf80b172fe4f47d1b5a81ce12438cdc2] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=71.8 K 2023-04-12 13:56:20,450 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3ad23180649e4f09a712578086113a65, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1681307776025 2023-04-12 13:56:20,450 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 6c9521f2cf39495b90e3613dc848ac7d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1681307776642 2023-04-12 13:56:20,450 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 4c2baa1173934e058670314032cf65c8, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681307775938 2023-04-12 13:56:20,450 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting f0ad97f06f5c4c75b20fd0112a9ebaeb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681307778282 2023-04-12 13:56:20,450 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ac654c086b094fac8c9f231a1301961f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1681307776025 2023-04-12 13:56:20,451 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting c1f25d2e449e436eb2f21edae1ea8a5f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1681307778625 2023-04-12 13:56:20,451 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ed24daacbb4b4a87aef35dab77629521, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1681307776642 2023-04-12 13:56:20,451 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 804d438e79ff492093af103a8650e97b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681307779354 2023-04-12 13:56:20,451 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f3f9b7f2385c4ecf8670f156863ba4f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681307778282 2023-04-12 13:56:20,451 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 147fdbd617be4c10b7f1f9818861be8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1681307778625 2023-04-12 13:56:20,452 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting bf80b172fe4f47d1b5a81ce12438cdc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681307779354 2023-04-12 13:56:20,464 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#564 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:20,465 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#563 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:20,484 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:20,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:20,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:20,487 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:20,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:20,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:20,488 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,488 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,489 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:20,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:20,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,489 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:20,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:20,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:20,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:20,492 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-12 13:56:20,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:20,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:20,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:20,492 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:20,493 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:20,493 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:20,507 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041244992b72b20247e1aebb8d63cf2b56f7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041244992b72b20247e1aebb8d63cf2b56f7 2023-04-12 13:56:20,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:20,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:20,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307780695, max timestamp=1681307780717Num uniques 859; 2023-04-12 13:56:20,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307780695, max timestamp=1681307780717Num uniques 859; 2023-04-12 13:56:20,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:20,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307780695, max timestamp=1681307780717Num uniques 859; 2023-04-12 13:56:20,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:20,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:20,794 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681307780774, max timestamp=1681307780792Num uniques 1018; 2023-04-12 13:56:20,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681307780774, max timestamp=1681307780792Num uniques 1018; 2023-04-12 13:56:20,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 13:56:20,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 13:56:20,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681307780774, max timestamp=1681307780792Num uniques 1018; 2023-04-12 13:56:20,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-12 13:56:20,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307840819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307840820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681307780815, max timestamp=1681307780824Num uniques 1027; 2023-04-12 13:56:20,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-12 13:56:20,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681307780815, max timestamp=1681307780824Num uniques 1027; 2023-04-12 13:56:20,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=14972, suffix heap size=134776,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=327 2023-04-12 13:56:20,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681307780817, max timestamp=1681307780824Num uniques 1027; 2023-04-12 13:56:20,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-12 13:56:20,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307840850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307840851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307840851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,884 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/f353df5cbab245b693807e778d2c91b9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f353df5cbab245b693807e778d2c91b9 2023-04-12 13:56:20,889 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/d00dc4c2dfc348f3b7454c3255d8ddf1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d00dc4c2dfc348f3b7454c3255d8ddf1 2023-04-12 13:56:20,890 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into f353df5cbab245b693807e778d2c91b9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:20,890 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:20,890 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=10, startTime=1681307780447; duration=0sec 2023-04-12 13:56:20,891 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:20,891 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-12 13:56:20,893 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73526 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-12 13:56:20,894 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction (all files) 2023-04-12 13:56:20,894 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:20,894 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/03f2ed7bacc9435f81a5b6b40e1f0472, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/58001a5e2cc8454783ddb9821a7377f7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f87dd1b87e5844f78c5871b8148d1574, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e839b7826218411fb9c3240af0ee4d58, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/721264f6e735469b9a4f726e85e7ab4d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c262ced90e404f399d37c29240fb86f2] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=71.8 K 2023-04-12 13:56:20,894 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 03f2ed7bacc9435f81a5b6b40e1f0472, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1681307775938 2023-04-12 13:56:20,895 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into d00dc4c2dfc348f3b7454c3255d8ddf1(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:20,895 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:20,895 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=10, startTime=1681307780447; duration=0sec 2023-04-12 13:56:20,895 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 58001a5e2cc8454783ddb9821a7377f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1681307776025 2023-04-12 13:56:20,895 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:20,895 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f87dd1b87e5844f78c5871b8148d1574, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1681307776642 2023-04-12 13:56:20,896 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e839b7826218411fb9c3240af0ee4d58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1681307778282 2023-04-12 13:56:20,896 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 721264f6e735469b9a4f726e85e7ab4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1681307778625 2023-04-12 13:56:20,896 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c262ced90e404f399d37c29240fb86f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681307779354 2023-04-12 13:56:20,907 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#566 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:20,913 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=612, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/3245a59bc2654fa8b584a7b64dbb6e1b 2023-04-12 13:56:20,916 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/b63899f348aa4745b6fa0fa921960a2e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/b63899f348aa4745b6fa0fa921960a2e 2023-04-12 13:56:20,921 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into b63899f348aa4745b6fa0fa921960a2e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:20,922 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:20,922 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=10, startTime=1681307780447; duration=0sec 2023-04-12 13:56:20,922 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:20,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307840928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307840928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307840956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307840956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:20,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:20,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307840956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307841132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307841133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307841159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307841159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307841160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,324 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/191de82725c945a394f289a2941b5148 2023-04-12 13:56:21,334 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/42a47ff7c2e245cfa6979a9635310d2c 2023-04-12 13:56:21,338 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/3245a59bc2654fa8b584a7b64dbb6e1b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3245a59bc2654fa8b584a7b64dbb6e1b 2023-04-12 13:56:21,341 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3245a59bc2654fa8b584a7b64dbb6e1b, entries=150, sequenceid=612, filesize=25.4 K 2023-04-12 13:56:21,342 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/191de82725c945a394f289a2941b5148 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/191de82725c945a394f289a2941b5148 2023-04-12 13:56:21,345 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/191de82725c945a394f289a2941b5148, entries=150, sequenceid=612, filesize=11.9 K 2023-04-12 13:56:21,346 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/42a47ff7c2e245cfa6979a9635310d2c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/42a47ff7c2e245cfa6979a9635310d2c 2023-04-12 13:56:21,349 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/42a47ff7c2e245cfa6979a9635310d2c, entries=150, sequenceid=612, filesize=11.9 K 2023-04-12 13:56:21,349 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=143.13 KB/146560 for e35088e2f9a6947abbfc447493169cd6 in 857ms, sequenceid=612, compaction requested=false 2023-04-12 13:56:21,349 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:21,349 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:21,351 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b065622[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:21,353 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:21,353 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,354 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:21,355 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,355 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:21,357 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,358 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:21,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:21,361 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:21,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:21,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:21,361 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:21,361 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:21,362 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,362 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,362 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,363 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,365 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,367 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:21,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:21,367 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,367 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:21,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:21,368 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:21,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:21,438 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=149.83 KB heapSize=418.84 KB 2023-04-12 13:56:21,438 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:21,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:21,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:21,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:21,439 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:21,439 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:21,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307841446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307841447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,456 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304124aacdb33488e4ec9967a991f53035fb4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124aacdb33488e4ec9967a991f53035fb4 2023-04-12 13:56:21,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307841460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307841463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307841463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,468 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:21,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:21,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:21,470 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:21,470 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:21,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:21,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,471 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,472 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,472 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:21,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:21,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:21,473 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:21,474 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,474 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:21,474 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:21,474 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:21,474 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:21,476 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:21,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c9b09da[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:21,477 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:21,477 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:21,479 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,479 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:21,479 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,480 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:21,483 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:21,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:21,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:21,483 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:21,483 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:21,485 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,485 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,485 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,486 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,487 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,491 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,491 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:21,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:21,491 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,491 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:21,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,492 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,492 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:21,492 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:21,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307841549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307841550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,592 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:21,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:21,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:21,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:21,596 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:21,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:21,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:21,597 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,597 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,598 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,598 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:21,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:21,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:21,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:21,599 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:21,600 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,600 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:21,600 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:21,600 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:21,600 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:21,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,602 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,602 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,602 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:21,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:21,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4da48b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:21,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:21,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,605 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,605 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:21,605 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:21,606 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:21,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,606 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:21,608 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,608 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:21,608 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,608 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:21,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:21,610 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:21,610 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:21,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:21,611 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:21,611 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:21,612 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,613 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,613 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,613 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,614 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,617 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,617 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:21,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:21,618 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:21,618 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:21,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:21,718 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:21,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:21,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:21,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:21,721 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:21,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:21,721 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:21,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,727 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,727 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,727 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,727 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:21,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:21,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:21,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:21,729 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:21,729 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:21,730 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,730 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:21,730 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:21,730 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:21,730 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:21,732 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,733 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:21,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff7192b[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:21,734 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:21,734 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,736 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:21,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:21,736 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:21,736 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,736 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:21,737 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:21,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,737 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,737 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:21,741 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:21,741 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:21,741 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:21,741 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:21,741 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:21,745 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,745 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,745 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,745 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,749 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307841753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:21,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307841753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,756 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,757 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,757 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:21,757 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:21,756 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,757 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:21,857 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:21,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:21,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:21,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:21,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:21,860 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:21,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:21,860 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:21,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,861 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=687, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/34834dd867924860a7add4ce9763cb1a 2023-04-12 13:56:21,862 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,862 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,863 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,863 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:21,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:21,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:21,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:21,865 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:21,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:21,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:21,865 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:21,866 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:21,867 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,867 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,868 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:21,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2669e9b1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:21,869 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:21,869 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/e9ede87c2aa0479f9415cd8910264262 2023-04-12 13:56:21,872 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:21,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,872 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:21,874 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:21,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:21,879 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:21,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:21,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:21,879 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:21,879 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:21,881 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,881 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,881 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:21,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/9947f535ea124ae08d7b69f4ef794305 2023-04-12 13:56:21,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:21,882 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,884 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/34834dd867924860a7add4ce9763cb1a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/34834dd867924860a7add4ce9763cb1a 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:21,887 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:21,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:21,887 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:21,888 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:21,888 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:21,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/34834dd867924860a7add4ce9763cb1a, entries=250, sequenceid=687, filesize=39.2 K 2023-04-12 13:56:21,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/e9ede87c2aa0479f9415cd8910264262 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e9ede87c2aa0479f9415cd8910264262 2023-04-12 13:56:21,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e9ede87c2aa0479f9415cd8910264262, entries=150, sequenceid=687, filesize=11.9 K 2023-04-12 13:56:21,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/9947f535ea124ae08d7b69f4ef794305 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9947f535ea124ae08d7b69f4ef794305 2023-04-12 13:56:21,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9947f535ea124ae08d7b69f4ef794305, entries=150, sequenceid=687, filesize=11.9 K 2023-04-12 13:56:21,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~156.54 KB/160300, heapSize ~439.42 KB/449968, currentSize=26.84 KB/27480 for e35088e2f9a6947abbfc447493169cd6 in 461ms, sequenceid=687, compaction requested=true 2023-04-12 13:56:21,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:21,899 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:21,899 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:21,899 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:21,899 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:21,899 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:21,900 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92836 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:21,900 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction (all files) 2023-04-12 13:56:21,900 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,900 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d00dc4c2dfc348f3b7454c3255d8ddf1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3245a59bc2654fa8b584a7b64dbb6e1b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/34834dd867924860a7add4ce9763cb1a] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=90.7 K 2023-04-12 13:56:21,900 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:21,900 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction (all files) 2023-04-12 13:56:21,900 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:21,901 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f353df5cbab245b693807e778d2c91b9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/191de82725c945a394f289a2941b5148, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e9ede87c2aa0479f9415cd8910264262] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=36.3 K 2023-04-12 13:56:21,901 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting d00dc4c2dfc348f3b7454c3255d8ddf1, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681307779354 2023-04-12 13:56:21,901 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 3245a59bc2654fa8b584a7b64dbb6e1b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1681307780010 2023-04-12 13:56:21,901 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f353df5cbab245b693807e778d2c91b9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681307779354 2023-04-12 13:56:21,901 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 34834dd867924860a7add4ce9763cb1a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681307780835 2023-04-12 13:56:21,902 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 191de82725c945a394f289a2941b5148, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1681307780010 2023-04-12 13:56:21,902 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e9ede87c2aa0479f9415cd8910264262, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681307780850 2023-04-12 13:56:21,908 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:21,913 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#572 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:21,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:21,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:21,988 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:21,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:21,992 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:21,992 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:21,992 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:21,992 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:21,993 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:21,993 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,000 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,000 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,000 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,000 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:22,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:22,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:22,002 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:22,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307781990, max timestamp=1681307782004Num uniques 859; 2023-04-12 13:56:22,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:22,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307781990, max timestamp=1681307782004Num uniques 859; 2023-04-12 13:56:22,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:22,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307781990, max timestamp=1681307782004Num uniques 859; 2023-04-12 13:56:22,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:22,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:22,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:22,007 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d8e17f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:22,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:22,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,010 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:22,011 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:22,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:22,014 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:22,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:22,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:22,014 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:22,014 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:22,020 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,020 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,020 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,021 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,023 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,025 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,026 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:22,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:22,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,026 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:22,026 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:22,026 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:22,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307842029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307842032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307842033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307842056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307842057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,127 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:22,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:22,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:22,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:22,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:22,131 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:22,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:22,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:22,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,132 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,133 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,133 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,134 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:22,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:22,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:22,135 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,135 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:22,136 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:22,136 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:22,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307842134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307842135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307842137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,140 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,141 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:22,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2717c53[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:22,142 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:22,142 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,144 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:22,144 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:22,144 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,144 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,145 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:22,148 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:22,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:22,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:22,148 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:22,148 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:22,150 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,150 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,150 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,151 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,152 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,154 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:22,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:22,155 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,155 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,155 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,155 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,155 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,155 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:22,255 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:22,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:22,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:22,259 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:22,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:22,259 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:22,260 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,260 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,261 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,261 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:22,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:22,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:22,263 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,263 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:22,263 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:22,263 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:22,263 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:22,265 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f81377[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:22,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:22,268 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,269 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,269 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:22,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,270 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,272 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,272 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:22,275 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:22,275 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:22,275 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:22,275 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:22,276 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:22,277 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,277 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,277 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,281 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,284 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,287 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,287 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:22,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:22,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,287 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:22,287 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:22,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:22,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,318 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/fb284f3f9b004b759c997067e436896c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/fb284f3f9b004b759c997067e436896c 2023-04-12 13:56:22,322 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into fb284f3f9b004b759c997067e436896c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:22,322 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:22,322 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=13, startTime=1681307781899; duration=0sec 2023-04-12 13:56:22,323 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:22,323 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-12 13:56:22,324 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-12 13:56:22,324 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction (all files) 2023-04-12 13:56:22,324 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,324 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/b63899f348aa4745b6fa0fa921960a2e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/42a47ff7c2e245cfa6979a9635310d2c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9947f535ea124ae08d7b69f4ef794305] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=36.3 K 2023-04-12 13:56:22,325 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting b63899f348aa4745b6fa0fa921960a2e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681307779354 2023-04-12 13:56:22,325 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 42a47ff7c2e245cfa6979a9635310d2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1681307780010 2023-04-12 13:56:22,325 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9947f535ea124ae08d7b69f4ef794305, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681307780850 2023-04-12 13:56:22,332 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#575 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:22,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307842339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307842340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307842341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,356 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/c8145ed65eed4bdcb6aa3083043083cf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c8145ed65eed4bdcb6aa3083043083cf 2023-04-12 13:56:22,360 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into c8145ed65eed4bdcb6aa3083043083cf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:22,360 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:22,361 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=13, startTime=1681307781899; duration=0sec 2023-04-12 13:56:22,361 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:22,388 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:22,389 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412b74581366e0643349d1bbedee0490d2f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b74581366e0643349d1bbedee0490d2f 2023-04-12 13:56:22,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:22,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:22,394 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:22,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:22,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:22,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,395 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,396 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,396 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:22,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,404 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:22,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:22,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:22,408 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,408 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:22,408 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:22,408 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:22,408 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:22,409 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:22,410 WARN [DataStreamer for file /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/7437133735aa48988312244fe1607d5c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-12 13:56:22,412 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=698, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/7437133735aa48988312244fe1607d5c 2023-04-12 13:56:22,412 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,412 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,413 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb55018[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:22,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:22,416 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,417 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:22,418 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:22,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,418 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,419 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:22,420 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/87244e7b605d4eaa8e49963a5dd4d8ed 2023-04-12 13:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:22,425 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:22,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:22,425 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:22,428 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:22,428 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,428 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:22,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:22,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:22,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:22,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:22,439 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:22,439 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:22,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/76c4b8663547447d97a992f917488837 2023-04-12 13:56:22,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:22,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/7437133735aa48988312244fe1607d5c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7437133735aa48988312244fe1607d5c 2023-04-12 13:56:22,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7437133735aa48988312244fe1607d5c, entries=300, sequenceid=698, filesize=46.1 K 2023-04-12 13:56:22,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/87244e7b605d4eaa8e49963a5dd4d8ed as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/87244e7b605d4eaa8e49963a5dd4d8ed 2023-04-12 13:56:22,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/87244e7b605d4eaa8e49963a5dd4d8ed, entries=150, sequenceid=698, filesize=11.9 K 2023-04-12 13:56:22,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/76c4b8663547447d97a992f917488837 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/76c4b8663547447d97a992f917488837 2023-04-12 13:56:22,459 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/76c4b8663547447d97a992f917488837, entries=150, sequenceid=698, filesize=11.9 K 2023-04-12 13:56:22,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for e35088e2f9a6947abbfc447493169cd6 in 488ms, sequenceid=698, compaction requested=false 2023-04-12 13:56:22,459 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:22,539 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:22,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:22,543 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:22,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:22,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:22,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,543 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,545 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:22,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:22,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:22,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:22,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:22,547 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:22,547 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:22,548 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-04-12 13:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:22,548 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:22,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:22,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:22,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307842582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307842582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307842642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307842642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307842647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307842687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307842694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,755 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/2ee5a90c86674370a27d1005d5de01af as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2ee5a90c86674370a27d1005d5de01af 2023-04-12 13:56:22,760 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into 2ee5a90c86674370a27d1005d5de01af(size=26.1 K), total size for store is 72.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:22,760 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:22,760 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=13, startTime=1681307781899; duration=0sec 2023-04-12 13:56:22,760 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:22,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307842891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:22,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307842897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:22,976 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304124e1a1acb709445f8834b3a9dc360b736 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124e1a1acb709445f8834b3a9dc360b736 2023-04-12 13:56:23,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307843144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307843147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307843151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307843196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307843202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,382 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=736, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/aa9dbb7ea3914c939cf092c9dbb64c7c 2023-04-12 13:56:23,395 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/1b5b8d5a26e1484fb509402c8c6cbef1 2023-04-12 13:56:23,405 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/529c2133edbb45e89e8da34df605ae0f 2023-04-12 13:56:23,409 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/aa9dbb7ea3914c939cf092c9dbb64c7c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/aa9dbb7ea3914c939cf092c9dbb64c7c 2023-04-12 13:56:23,413 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/aa9dbb7ea3914c939cf092c9dbb64c7c, entries=150, sequenceid=736, filesize=25.4 K 2023-04-12 13:56:23,413 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/1b5b8d5a26e1484fb509402c8c6cbef1 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/1b5b8d5a26e1484fb509402c8c6cbef1 2023-04-12 13:56:23,417 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/1b5b8d5a26e1484fb509402c8c6cbef1, entries=150, sequenceid=736, filesize=11.9 K 2023-04-12 13:56:23,417 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/529c2133edbb45e89e8da34df605ae0f as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/529c2133edbb45e89e8da34df605ae0f 2023-04-12 13:56:23,420 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/529c2133edbb45e89e8da34df605ae0f, entries=150, sequenceid=736, filesize=11.9 K 2023-04-12 13:56:23,421 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=53.67 KB/54960 for e35088e2f9a6947abbfc447493169cd6 in 873ms, sequenceid=736, compaction requested=true 2023-04-12 13:56:23,421 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:23,421 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:23,421 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,423 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:23,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e391e8[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:23,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:23,425 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,426 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:23,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:23,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,427 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:23,433 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:23,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:23,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:23,436 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:23,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:23,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:23,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:23,436 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:23,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,437 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:23,438 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:23,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:23,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:23,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:23,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:23,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:23,444 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:23,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:23,444 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:23,444 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:23,444 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:23,544 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:23,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:23,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:23,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:23,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:23,548 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:23,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:23,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:23,548 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:23,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,549 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:23,552 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,552 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:23,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:23,553 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,553 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:23,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:23,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:23,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:23,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:23,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:23,554 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:23,555 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:23,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:23,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:23,555 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:56:23,555 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:23,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:23,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:23,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:23,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:23,556 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:23,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:23,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:23,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307783713, max timestamp=1681307783734Num uniques 859; 2023-04-12 13:56:23,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:23,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307783713, max timestamp=1681307783734Num uniques 859; 2023-04-12 13:56:23,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307783713, max timestamp=1681307783734Num uniques 859; 2023-04-12 13:56:23,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:23,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:23,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307843762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307843762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307843868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307843868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:23,974 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304122d1c0ecacdd345ae941316cd3bd27899 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122d1c0ecacdd345ae941316cd3bd27899 2023-04-12 13:56:23,979 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=748, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/2846ed80b19c4bf782422bb34839f738 2023-04-12 13:56:23,993 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ef7c4b439a744e398acfe9d031f7b82e 2023-04-12 13:56:24,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307844072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307844072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307844147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307844153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307844155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307844375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307844376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,407 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/9260b20268cf4c8091c8b18efc046d46 2023-04-12 13:56:24,411 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/2846ed80b19c4bf782422bb34839f738 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2846ed80b19c4bf782422bb34839f738 2023-04-12 13:56:24,415 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2846ed80b19c4bf782422bb34839f738, entries=100, sequenceid=748, filesize=18.6 K 2023-04-12 13:56:24,415 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ef7c4b439a744e398acfe9d031f7b82e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef7c4b439a744e398acfe9d031f7b82e 2023-04-12 13:56:24,419 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef7c4b439a744e398acfe9d031f7b82e, entries=100, sequenceid=748, filesize=9.5 K 2023-04-12 13:56:24,419 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/9260b20268cf4c8091c8b18efc046d46 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9260b20268cf4c8091c8b18efc046d46 2023-04-12 13:56:24,422 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9260b20268cf4c8091c8b18efc046d46, entries=100, sequenceid=748, filesize=9.5 K 2023-04-12 13:56:24,423 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e35088e2f9a6947abbfc447493169cd6 in 868ms, sequenceid=748, compaction requested=true 2023-04-12 13:56:24,423 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:24,423 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,425 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:24,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5079fd63[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:24,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:24,426 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,429 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:24,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,429 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:24,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:24,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:24,430 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,430 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:24,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:24,433 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:24,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:24,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:24,433 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:24,433 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:24,435 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,435 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,435 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:24,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:24,436 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:24,438 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:24,441 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:24,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:24,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:24,441 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,441 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:24,441 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:24,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:24,442 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:24,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:24,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:24,542 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:24,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:24,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:24,545 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,545 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:24,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:24,546 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:24,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,547 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:24,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:24,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:24,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:24,549 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:24,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:24,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307844894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:24,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307844898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:24,961 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041266cbaaddb42940f99ae8819a7f1d8010 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041266cbaaddb42940f99ae8819a7f1d8010 2023-04-12 13:56:24,966 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=783, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/203887184dad48589d560e0117e303d3 2023-04-12 13:56:25,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:25,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307844999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:25,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307845002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:25,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307845203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307845206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,377 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/3273c2cb89ae47d5970999d018d95952 2023-04-12 13:56:25,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:25,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307845507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:25,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307845510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,786 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/3ae64a42aab847a899ae3792ae60cbfd 2023-04-12 13:56:25,791 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/203887184dad48589d560e0117e303d3 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/203887184dad48589d560e0117e303d3 2023-04-12 13:56:25,795 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/203887184dad48589d560e0117e303d3, entries=150, sequenceid=783, filesize=25.4 K 2023-04-12 13:56:25,796 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/3273c2cb89ae47d5970999d018d95952 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/3273c2cb89ae47d5970999d018d95952 2023-04-12 13:56:25,799 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/3273c2cb89ae47d5970999d018d95952, entries=150, sequenceid=783, filesize=11.9 K 2023-04-12 13:56:25,800 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/3ae64a42aab847a899ae3792ae60cbfd as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/3ae64a42aab847a899ae3792ae60cbfd 2023-04-12 13:56:25,803 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/3ae64a42aab847a899ae3792ae60cbfd, entries=150, sequenceid=783, filesize=11.9 K 2023-04-12 13:56:25,804 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e35088e2f9a6947abbfc447493169cd6 in 1254ms, sequenceid=783, compaction requested=true 2023-04-12 13:56:25,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:25,804 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:25,804 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:25,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,806 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:25,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:25,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ef2fb0[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:25,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:25,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,809 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:25,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:25,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:25,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:25,811 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:25,814 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:25,814 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:25,814 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:25,814 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:25,814 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:25,817 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,817 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,817 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:25,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,818 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:25,820 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:25,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:25,822 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:25,822 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:25,823 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:25,823 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,823 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,823 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:25,823 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:25,823 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,823 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:25,923 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:25,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:25,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:25,927 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:25,927 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:25,927 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:25,927 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:25,928 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,928 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:25,929 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:25,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:25,929 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:25,930 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:25,930 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:25,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:25,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:25,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:25,931 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:25,931 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:25,931 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:56:25,931 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:25,931 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:25,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:25,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:25,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:25,932 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:26,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:26,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:26,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307786038, max timestamp=1681307786048Num uniques 859; 2023-04-12 13:56:26,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:26,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307786038, max timestamp=1681307786048Num uniques 859; 2023-04-12 13:56:26,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:26,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681307786038, max timestamp=1681307786048Num uniques 859; 2023-04-12 13:56:26,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:26,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307846090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307846093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307846156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,159 DEBUG [Thread-3700] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:26,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307846161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,165 DEBUG [Thread-3702] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:26,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307846175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,179 DEBUG [Thread-3698] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:26,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307846194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307846197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,353 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304128f2deb01819f4eab9246e2dfecfc3324 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128f2deb01819f4eab9246e2dfecfc3324 2023-04-12 13:56:26,358 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=794, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/4f50f971b15549b1b0b8ad4847b0a9a0 2023-04-12 13:56:26,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307846400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307846406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307846705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:26,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307846712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,767 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ef6b924385f84b409ba0181b36574a48 2023-04-12 13:56:26,778 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=794 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/7490a41ff9b14eecaa55ded26e41d260 2023-04-12 13:56:26,781 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/4f50f971b15549b1b0b8ad4847b0a9a0 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4f50f971b15549b1b0b8ad4847b0a9a0 2023-04-12 13:56:26,784 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4f50f971b15549b1b0b8ad4847b0a9a0, entries=150, sequenceid=794, filesize=25.4 K 2023-04-12 13:56:26,784 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ef6b924385f84b409ba0181b36574a48 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef6b924385f84b409ba0181b36574a48 2023-04-12 13:56:26,787 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef6b924385f84b409ba0181b36574a48, entries=150, sequenceid=794, filesize=11.9 K 2023-04-12 13:56:26,788 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/7490a41ff9b14eecaa55ded26e41d260 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7490a41ff9b14eecaa55ded26e41d260 2023-04-12 13:56:26,791 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7490a41ff9b14eecaa55ded26e41d260, entries=150, sequenceid=794, filesize=11.9 K 2023-04-12 13:56:26,791 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e35088e2f9a6947abbfc447493169cd6 in 860ms, sequenceid=794, compaction requested=true 2023-04-12 13:56:26,792 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:26,792 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:26,792 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:26,793 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,793 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:26,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,794 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:26,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:26,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:26,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@532c8e85[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:26,795 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:26,795 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,797 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:26,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:26,797 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:26,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,798 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:26,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:26,799 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:26,799 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:26,802 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:26,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:26,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:26,802 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:26,802 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:26,805 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,806 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,806 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:26,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:26,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,807 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,808 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:26,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:26,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:26,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:26,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:26,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:26,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:26,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:26,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:26,812 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,812 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:26,812 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,812 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:26,812 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:26,812 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,912 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:26,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:26,916 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:26,916 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:26,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:26,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:26,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,917 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:26,918 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,918 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:26,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:26,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:26,919 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:26,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:26,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:26,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:26,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:26,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:26,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:26,920 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:26,920 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:26,921 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:26,921 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:26,937 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a6c4f59384004ae0811a0696d4a5aa1a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a6c4f59384004ae0811a0696d4a5aa1a 2023-04-12 13:56:26,943 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=829, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/641d801404954837bd1b3e81775697a6 2023-04-12 13:56:26,955 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/e2e4096e9de4446bb70167e33c81a178 2023-04-12 13:56:27,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:27,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:27,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307847229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307847231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307847332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307847338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,368 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/1095d91620874c66be5bb9f14ae3fb67 2023-04-12 13:56:27,372 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/641d801404954837bd1b3e81775697a6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/641d801404954837bd1b3e81775697a6 2023-04-12 13:56:27,375 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/641d801404954837bd1b3e81775697a6, entries=150, sequenceid=829, filesize=25.4 K 2023-04-12 13:56:27,376 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/e2e4096e9de4446bb70167e33c81a178 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e2e4096e9de4446bb70167e33c81a178 2023-04-12 13:56:27,379 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e2e4096e9de4446bb70167e33c81a178, entries=150, sequenceid=829, filesize=11.9 K 2023-04-12 13:56:27,379 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/1095d91620874c66be5bb9f14ae3fb67 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1095d91620874c66be5bb9f14ae3fb67 2023-04-12 13:56:27,382 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1095d91620874c66be5bb9f14ae3fb67, entries=150, sequenceid=829, filesize=11.9 K 2023-04-12 13:56:27,383 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e35088e2f9a6947abbfc447493169cd6 in 462ms, sequenceid=829, compaction requested=true 2023-04-12 13:56:27,383 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:27,383 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:27,386 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,387 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168529d9[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:27,388 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:27,388 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,389 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:27,390 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,390 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:27,390 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:27,391 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:27,394 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:27,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:27,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:27,394 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:27,394 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:27,396 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,396 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:27,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,396 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:27,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,397 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,398 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:27,401 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:27,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:27,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,401 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:27,401 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,401 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:27,501 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:27,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:27,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:27,505 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:27,505 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:27,505 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:27,505 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:27,505 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,506 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:27,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,507 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:27,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:27,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,507 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:27,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:27,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:27,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:27,508 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:27,509 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:27,509 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:27,509 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:27,509 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:27,510 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:27,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307787576, max timestamp=1681307787586Num uniques 859; 2023-04-12 13:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307787576, max timestamp=1681307787586Num uniques 859; 2023-04-12 13:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681307787576, max timestamp=1681307787586Num uniques 859; 2023-04-12 13:56:27,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-12 13:56:27,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307847622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307847624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307847727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307847728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,929 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304125ba9048260b84978a913d2f28be6828b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125ba9048260b84978a913d2f28be6828b 2023-04-12 13:56:27,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307847931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307847932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,937 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=840, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/e0f8c09d8a0645e6bff60bb1e463938c 2023-04-12 13:56:27,957 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/624a1f349e074f8aa204a6742b96e8d8 2023-04-12 13:56:27,970 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/372bf1b75ce045c3824f502896c48309 2023-04-12 13:56:27,974 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/e0f8c09d8a0645e6bff60bb1e463938c as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e0f8c09d8a0645e6bff60bb1e463938c 2023-04-12 13:56:27,977 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e0f8c09d8a0645e6bff60bb1e463938c, entries=150, sequenceid=840, filesize=25.4 K 2023-04-12 13:56:27,978 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/624a1f349e074f8aa204a6742b96e8d8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/624a1f349e074f8aa204a6742b96e8d8 2023-04-12 13:56:27,981 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/624a1f349e074f8aa204a6742b96e8d8, entries=150, sequenceid=840, filesize=11.9 K 2023-04-12 13:56:27,982 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/372bf1b75ce045c3824f502896c48309 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/372bf1b75ce045c3824f502896c48309 2023-04-12 13:56:27,985 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/372bf1b75ce045c3824f502896c48309, entries=150, sequenceid=840, filesize=11.9 K 2023-04-12 13:56:27,985 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e35088e2f9a6947abbfc447493169cd6 in 476ms, sequenceid=840, compaction requested=true 2023-04-12 13:56:27,986 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:27,986 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:27,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:27,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:27,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:27,986 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:27,987 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,987 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,988 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9085c8f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:27,990 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:27,990 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,991 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:27,991 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,991 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:27,991 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:27,998 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:27,998 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:27,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:27,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:28,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:28,002 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:28,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:28,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:28,002 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:28,002 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:28,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:28,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,004 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:28,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:28,004 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,006 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:28,008 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:28,009 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:28,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:28,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:28,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:28,009 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:28,009 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:28,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:28,009 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:28,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:28,109 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:28,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:28,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:28,113 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:28,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:28,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:28,113 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:28,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,114 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,115 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:28,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:28,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:28,115 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:28,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,116 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:28,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:28,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:28,117 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:28,117 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:28,117 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:28,117 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:28,117 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:28,117 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:28,117 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:28,118 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-12 13:56:28,118 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:28,118 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:28,118 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:28,118 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:28,118 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:28,118 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:28,131 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041256e7a1af237149c6850464c973b83af4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041256e7a1af237149c6850464c973b83af4 2023-04-12 13:56:28,137 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=875, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/4424d5d6940e41eab6bf6e5d800e398b 2023-04-12 13:56:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:28,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:28,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307848259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307848260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307848364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307848365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,554 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/22d4f2e8e4c847ae8de437889d23a410 2023-04-12 13:56:28,565 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/e9aeeabd41aa470ca50562a1d0fa44e8 2023-04-12 13:56:28,569 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/4424d5d6940e41eab6bf6e5d800e398b as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4424d5d6940e41eab6bf6e5d800e398b 2023-04-12 13:56:28,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307848567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307848568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,572 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4424d5d6940e41eab6bf6e5d800e398b, entries=150, sequenceid=875, filesize=25.4 K 2023-04-12 13:56:28,573 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/22d4f2e8e4c847ae8de437889d23a410 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/22d4f2e8e4c847ae8de437889d23a410 2023-04-12 13:56:28,576 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/22d4f2e8e4c847ae8de437889d23a410, entries=150, sequenceid=875, filesize=11.9 K 2023-04-12 13:56:28,577 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/e9aeeabd41aa470ca50562a1d0fa44e8 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e9aeeabd41aa470ca50562a1d0fa44e8 2023-04-12 13:56:28,580 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e9aeeabd41aa470ca50562a1d0fa44e8, entries=150, sequenceid=875, filesize=11.9 K 2023-04-12 13:56:28,581 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e35088e2f9a6947abbfc447493169cd6 in 463ms, sequenceid=875, compaction requested=true 2023-04-12 13:56:28,581 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:28,581 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:28,584 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:28,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:28,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c660b1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:28,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:28,586 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,587 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,588 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:28,588 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:28,588 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:28,588 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,588 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:28,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:28,589 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:28,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:28,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:28,592 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:28,592 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:28,592 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:28,592 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:28,592 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:28,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,594 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:28,595 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,596 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:28,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:28,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:28,602 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:28,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:28,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:28,602 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,603 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:28,603 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,603 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:28,603 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:28,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:28,703 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:28,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:28,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:28,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:28,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:28,707 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:28,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:28,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:28,707 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:28,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,708 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,709 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:28,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:28,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,710 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:28,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:28,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:28,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:28,711 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:28,711 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:28,711 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:28,711 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:28,711 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:28,711 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:28,711 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:28,722 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041217ebc7dcaf16400d8076b0576aab22eb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041217ebc7dcaf16400d8076b0576aab22eb 2023-04-12 13:56:28,727 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=886, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/ae847923fe004ba1997313af984c2ee4 2023-04-12 13:56:28,735 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/0df4f54badbd44519b914d7bfde7e7a4 2023-04-12 13:56:28,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:28,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:28,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307848916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:28,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307788903, max timestamp=1681307788920Num uniques 859; 2023-04-12 13:56:28,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:28,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307788903, max timestamp=1681307788920Num uniques 859; 2023-04-12 13:56:28,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:28,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307788903, max timestamp=1681307788920Num uniques 859; 2023-04-12 13:56:28,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:29,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307848996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307849022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307849101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,144 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/fbd769cb8adf450c832b9a9bcc0c785e 2023-04-12 13:56:29,148 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/ae847923fe004ba1997313af984c2ee4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/ae847923fe004ba1997313af984c2ee4 2023-04-12 13:56:29,151 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/ae847923fe004ba1997313af984c2ee4, entries=150, sequenceid=886, filesize=25.4 K 2023-04-12 13:56:29,152 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/0df4f54badbd44519b914d7bfde7e7a4 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/0df4f54badbd44519b914d7bfde7e7a4 2023-04-12 13:56:29,156 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/0df4f54badbd44519b914d7bfde7e7a4, entries=150, sequenceid=886, filesize=11.9 K 2023-04-12 13:56:29,156 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/fbd769cb8adf450c832b9a9bcc0c785e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fbd769cb8adf450c832b9a9bcc0c785e 2023-04-12 13:56:29,159 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fbd769cb8adf450c832b9a9bcc0c785e, entries=150, sequenceid=886, filesize=11.9 K 2023-04-12 13:56:29,159 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e35088e2f9a6947abbfc447493169cd6 in 448ms, sequenceid=886, compaction requested=true 2023-04-12 13:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:29,160 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:29,161 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,161 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:29,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6229942a[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:29,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:29,163 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,165 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:29,168 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:29,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:29,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:29,169 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:29,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:29,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:29,169 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:29,169 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:29,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,170 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,171 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,172 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,174 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:29,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:29,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:29,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,174 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,175 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:29,175 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:29,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,175 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:29,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-12 13:56:29,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:29,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:29,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:29,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:29,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:29,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-12 13:56:29,241 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304129fb6a34ebc1f44928448f45ad81ea109 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129fb6a34ebc1f44928448f45ad81ea109 2023-04-12 13:56:29,254 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=923, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/e3930e70ad1e40fba88341e2765a9719 2023-04-12 13:56:29,275 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:29,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:29,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307849271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:29,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:29,287 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:29,287 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:29,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:29,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/7b2ef6aa5292475c964bff2f338ebc50 2023-04-12 13:56:29,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,288 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,294 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,294 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:29,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:29,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:29,296 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,296 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:29,296 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:29,296 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:29,296 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:29,300 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:29,301 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:29,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ddda257[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:29,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:29,304 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,305 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:29,305 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:29,307 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:29,311 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:29,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:29,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:29,311 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:29,311 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:29,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307849307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,313 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,314 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,314 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,315 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,317 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,318 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,318 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,318 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,318 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,318 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,318 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:29,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:29,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,318 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,318 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:29,318 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:29,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:29,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307849377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,418 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:29,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:29,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:29,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:29,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:29,422 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:29,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:29,422 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:29,423 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,423 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,424 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,424 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:29,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:29,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:29,426 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,426 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:29,426 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:29,426 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:29,428 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167a7147[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:29,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:29,430 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,432 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:29,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,432 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:29,434 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,434 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:29,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:29,437 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:29,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:29,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:29,437 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:29,437 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:29,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,439 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,439 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,440 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:29,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:29,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:29,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,449 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,449 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,450 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,450 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:29,450 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:29,550 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:29,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:29,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:29,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:29,554 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:29,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:29,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:29,554 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,555 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,556 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,556 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:29,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:29,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:29,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:29,558 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,558 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:29,558 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:29,558 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,559 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,560 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:29,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70df48d1[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:29,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:29,561 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,567 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:29,567 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:29,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,567 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:29,568 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,568 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,569 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:29,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:29,571 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:29,571 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:29,571 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:29,571 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:29,571 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:29,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,573 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,574 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,575 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,577 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:29,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:29,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,577 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,578 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:29,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:29,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,578 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:29,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,578 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307849581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307849614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,678 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:29,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:29,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:29,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:29,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:29,682 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:29,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:29,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:29,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,682 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,685 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,685 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:29,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:29,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:29,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:29,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:29,687 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:29,689 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,689 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:29,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef91a24[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:29,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:29,691 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,692 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:29,692 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:29,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,693 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:29,694 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,694 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:29,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:29,696 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:29,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:29,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:29,697 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:29,697 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:29,700 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,700 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,700 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:29,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:29,701 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,702 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/862818e76e104dfab40d3b3344713302 2023-04-12 13:56:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:29,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:29,705 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,705 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:29,705 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:29,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:29,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,705 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:29,706 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:29,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:29,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/e3930e70ad1e40fba88341e2765a9719 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e3930e70ad1e40fba88341e2765a9719 2023-04-12 13:56:29,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e3930e70ad1e40fba88341e2765a9719, entries=200, sequenceid=923, filesize=32.3 K 2023-04-12 13:56:29,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/7b2ef6aa5292475c964bff2f338ebc50 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/7b2ef6aa5292475c964bff2f338ebc50 2023-04-12 13:56:29,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/7b2ef6aa5292475c964bff2f338ebc50, entries=150, sequenceid=923, filesize=11.9 K 2023-04-12 13:56:29,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/862818e76e104dfab40d3b3344713302 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/862818e76e104dfab40d3b3344713302 2023-04-12 13:56:29,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/862818e76e104dfab40d3b3344713302, entries=150, sequenceid=923, filesize=11.9 K 2023-04-12 13:56:29,718 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for e35088e2f9a6947abbfc447493169cd6 in 490ms, sequenceid=923, compaction requested=true 2023-04-12 13:56:29,718 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:29,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:29,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:29,718 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-04-12 13:56:29,718 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-04-12 13:56:29,718 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 275252 starting at candidate #0 after considering 44 permutations with 44 in ratio 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119416 starting at candidate #1 after considering 44 permutations with 44 in ratio 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction 2023-04-12 13:56:29,721 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,721 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,721 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2ee5a90c86674370a27d1005d5de01af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7437133735aa48988312244fe1607d5c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/aa9dbb7ea3914c939cf092c9dbb64c7c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2846ed80b19c4bf782422bb34839f738, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/203887184dad48589d560e0117e303d3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4f50f971b15549b1b0b8ad4847b0a9a0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/641d801404954837bd1b3e81775697a6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e0f8c09d8a0645e6bff60bb1e463938c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4424d5d6940e41eab6bf6e5d800e398b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/ae847923fe004ba1997313af984c2ee4] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=268.8 K 2023-04-12 13:56:29,721 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/87244e7b605d4eaa8e49963a5dd4d8ed, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/1b5b8d5a26e1484fb509402c8c6cbef1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef7c4b439a744e398acfe9d031f7b82e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/3273c2cb89ae47d5970999d018d95952, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef6b924385f84b409ba0181b36574a48, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e2e4096e9de4446bb70167e33c81a178, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/624a1f349e074f8aa204a6742b96e8d8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/22d4f2e8e4c847ae8de437889d23a410, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/0df4f54badbd44519b914d7bfde7e7a4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/7b2ef6aa5292475c964bff2f338ebc50] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=116.6 K 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2ee5a90c86674370a27d1005d5de01af, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=687 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 87244e7b605d4eaa8e49963a5dd4d8ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 7437133735aa48988312244fe1607d5c, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=698 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1b5b8d5a26e1484fb509402c8c6cbef1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=736 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting aa9dbb7ea3914c939cf092c9dbb64c7c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=736 2023-04-12 13:56:29,721 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ef7c4b439a744e398acfe9d031f7b82e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=748 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 2846ed80b19c4bf782422bb34839f738, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=748 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3273c2cb89ae47d5970999d018d95952, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 203887184dad48589d560e0117e303d3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=783 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting ef6b924385f84b409ba0181b36574a48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4f50f971b15549b1b0b8ad4847b0a9a0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=794 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e2e4096e9de4446bb70167e33c81a178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 641d801404954837bd1b3e81775697a6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=829 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 624a1f349e074f8aa204a6742b96e8d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e0f8c09d8a0645e6bff60bb1e463938c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=840 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 22d4f2e8e4c847ae8de437889d23a410, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 4424d5d6940e41eab6bf6e5d800e398b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=875 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 0df4f54badbd44519b914d7bfde7e7a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=886 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7b2ef6aa5292475c964bff2f338ebc50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923 2023-04-12 13:56:29,722 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting ae847923fe004ba1997313af984c2ee4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=886 2023-04-12 13:56:29,740 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#606 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:29,745 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#605 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:29,750 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/06d1d56d65b549de93969eb3e88c375e as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/06d1d56d65b549de93969eb3e88c375e 2023-04-12 13:56:29,754 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into 06d1d56d65b549de93969eb3e88c375e(size=12.2 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:29,754 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:29,754 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=5, startTime=1681307789718; duration=0sec 2023-04-12 13:56:29,754 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:29,755 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119416 starting at candidate #1 after considering 44 permutations with 44 in ratio 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction 2023-04-12 13:56:29,758 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,758 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/76c4b8663547447d97a992f917488837, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/529c2133edbb45e89e8da34df605ae0f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9260b20268cf4c8091c8b18efc046d46, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/3ae64a42aab847a899ae3792ae60cbfd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7490a41ff9b14eecaa55ded26e41d260, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1095d91620874c66be5bb9f14ae3fb67, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/372bf1b75ce045c3824f502896c48309, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e9aeeabd41aa470ca50562a1d0fa44e8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fbd769cb8adf450c832b9a9bcc0c785e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/862818e76e104dfab40d3b3344713302] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=116.6 K 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 76c4b8663547447d97a992f917488837, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 529c2133edbb45e89e8da34df605ae0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=736 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 9260b20268cf4c8091c8b18efc046d46, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=748 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 3ae64a42aab847a899ae3792ae60cbfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 7490a41ff9b14eecaa55ded26e41d260, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=794 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 1095d91620874c66be5bb9f14ae3fb67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829 2023-04-12 13:56:29,758 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 372bf1b75ce045c3824f502896c48309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840 2023-04-12 13:56:29,759 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting e9aeeabd41aa470ca50562a1d0fa44e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875 2023-04-12 13:56:29,759 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting fbd769cb8adf450c832b9a9bcc0c785e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=886 2023-04-12 13:56:29,759 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 862818e76e104dfab40d3b3344713302, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923 2023-04-12 13:56:29,759 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/5b9e5f43c84e43da81162ad0371d5b4d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5b9e5f43c84e43da81162ad0371d5b4d 2023-04-12 13:56:29,764 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into 5b9e5f43c84e43da81162ad0371d5b4d(size=26.5 K), total size for store is 58.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:29,764 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:29,764 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=5, startTime=1681307789718; duration=0sec 2023-04-12 13:56:29,764 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:29,780 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#607 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:29,789 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/a4b17fcb67ad4413ac7f90612c957d88 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/a4b17fcb67ad4413ac7f90612c957d88 2023-04-12 13:56:29,793 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into a4b17fcb67ad4413ac7f90612c957d88(size=12.2 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:29,793 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:29,793 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=5, startTime=1681307789718; duration=0sec 2023-04-12 13:56:29,794 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:29,806 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:29,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:29,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:29,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:29,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:29,809 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:29,809 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:29,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,810 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,811 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:29,811 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,812 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:29,812 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:29,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:29,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:29,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:29,813 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:29,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:29,814 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:29,814 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:29,814 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:29,814 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:29,814 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:29,814 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:29,814 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:29,815 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:29,815 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:29,815 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:29,829 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230412a301df2b56134abb944b56712ef5e344 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a301df2b56134abb944b56712ef5e344 2023-04-12 13:56:29,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:29,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307789913, max timestamp=1681307789931Num uniques 859; 2023-04-12 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307789913, max timestamp=1681307789931Num uniques 859; 2023-04-12 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681307789913, max timestamp=1681307789931Num uniques 859; 2023-04-12 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-12 13:56:29,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:29,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307849982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307850085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307850118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44752 deadline: 1681307850164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,167 DEBUG [Thread-3700] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:30,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44732 deadline: 1681307850182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,185 DEBUG [Thread-3702] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:30,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44798 deadline: 1681307850197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,201 DEBUG [Thread-3698] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., hostname=jenkins-hbase4.apache.org,35663,1681307616979, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 13:56:30,236 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=936, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/dd9f696a6dd6417283de14537d493dd5 2023-04-12 13:56:30,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307850288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:30,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307850591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:30,644 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/6cecd8d3e0fa40d9a8b2da6937f1a040 2023-04-12 13:56:31,054 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/c87b8430af7c41d199ddac1bdc9ab94d 2023-04-12 13:56:31,058 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/dd9f696a6dd6417283de14537d493dd5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/dd9f696a6dd6417283de14537d493dd5 2023-04-12 13:56:31,062 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/dd9f696a6dd6417283de14537d493dd5, entries=150, sequenceid=936, filesize=25.4 K 2023-04-12 13:56:31,063 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/6cecd8d3e0fa40d9a8b2da6937f1a040 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6cecd8d3e0fa40d9a8b2da6937f1a040 2023-04-12 13:56:31,065 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6cecd8d3e0fa40d9a8b2da6937f1a040, entries=150, sequenceid=936, filesize=11.9 K 2023-04-12 13:56:31,066 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/c87b8430af7c41d199ddac1bdc9ab94d as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c87b8430af7c41d199ddac1bdc9ab94d 2023-04-12 13:56:31,069 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c87b8430af7c41d199ddac1bdc9ab94d, entries=150, sequenceid=936, filesize=11.9 K 2023-04-12 13:56:31,069 INFO [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e35088e2f9a6947abbfc447493169cd6 in 1255ms, sequenceid=936, compaction requested=true 2023-04-12 13:56:31,070 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:31,070 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,071 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,071 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,072 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,072 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e06c4b6[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,073 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,073 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,075 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,075 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,075 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,077 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,077 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,081 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,081 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,081 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,081 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,081 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,083 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,083 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,083 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,084 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,086 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,088 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,088 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,088 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,088 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,088 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,089 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,089 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,089 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,089 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,089 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,089 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,089 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:31,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:31,098 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-12 13:56:31,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:31,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:31,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:31,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:31,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:31,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-12 13:56:31,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307851117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:31,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44766 deadline: 1681307851121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,129 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304121d7993aef228495b9d86ec183616db81 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121d7993aef228495b9d86ec183616db81 2023-04-12 13:56:31,189 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:31,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:31,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:31,193 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:31,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:31,193 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:31,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,194 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,195 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,195 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,196 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:31,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:31,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:31,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:31,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:31,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:31,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:31,197 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:31,197 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,197 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:31,197 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:31,198 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,198 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,203 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,203 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,204 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb37078[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,206 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,206 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,207 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,207 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,207 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,207 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,209 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,209 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,209 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,209 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,215 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,215 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,215 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,215 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,215 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,218 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,218 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,218 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,219 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,221 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35663] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307851220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,224 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,224 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,225 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,225 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:31,325 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:31,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:31,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:31,328 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:31,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:31,328 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:31,329 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,329 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,330 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,330 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,330 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,331 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:31,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:31,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:31,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:31,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:31,332 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,332 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:31,332 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:31,332 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,332 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,334 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,334 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,335 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21202c1f[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,336 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,336 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,338 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,339 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,339 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,340 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,342 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,342 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,342 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,342 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,342 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,344 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,344 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,344 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,346 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,347 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,349 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,350 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,349 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,350 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,350 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:31,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307851424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,450 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:31,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:31,454 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:31,454 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,454 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:31,454 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:31,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,455 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,458 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,458 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:31,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:31,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:31,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,459 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:31,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:31,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:31,460 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,460 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,462 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,462 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@379e3a25[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,464 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,464 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,465 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,466 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,466 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,466 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,468 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,468 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,468 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,468 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,470 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,470 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,470 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,470 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,470 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,473 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,473 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,474 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,475 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,476 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,478 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,479 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:31,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,479 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,479 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,480 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,547 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=974, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/7d67b3b25fd348548324bf1c7431c195 2023-04-12 13:56:31,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/f81eeb248cbe40b882b0bc3290e2150a 2023-04-12 13:56:31,580 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:31,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:31,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:31,583 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:31,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:31,583 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:31,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,584 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,585 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,585 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:31,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:31,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:31,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:31,587 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,587 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:31,587 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:31,587 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,588 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,589 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe57b14[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,590 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,595 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,595 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,595 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,595 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,595 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,595 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,596 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,599 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,599 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,599 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,599 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,599 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,601 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,601 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,602 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,604 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,604 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,604 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,604 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,604 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-12 13:56:31,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,604 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,605 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,605 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,605 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,605 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,605 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,605 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,705 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:31,709 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:31,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:31,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:31,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,709 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,714 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,714 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:31,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:31,715 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:31,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:31,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:31,716 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,716 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,718 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,718 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f65e620[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,720 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,720 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,722 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,722 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,724 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,727 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,727 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,727 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,727 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,727 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-12 13:56:31,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35663] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44784 deadline: 1681307851728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,730 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,730 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,731 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,732 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,733 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,737 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,738 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,738 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,737 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,738 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,738 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,838 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 13:56:31,840 DEBUG [Thread-3707] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:58527 2023-04-12 13:56:31,841 DEBUG [Thread-3707] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:31,841 DEBUG [Thread-3713] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28164190 to 127.0.0.1:58527 2023-04-12 13:56:31,841 DEBUG [Thread-3713] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:31,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-12 13:56:31,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-12 13:56:31,842 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-12 13:56:31,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,842 DEBUG [Thread-3709] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:58527 2023-04-12 13:56:31,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-12 13:56:31,842 DEBUG [Thread-3709] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:31,842 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-12 13:56:31,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,843 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,843 DEBUG [Thread-3705] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:58527 2023-04-12 13:56:31,843 DEBUG [Thread-3705] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:31,844 DEBUG [Thread-3711] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:58527 2023-04-12 13:56:31,844 DEBUG [Thread-3711] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:31,846 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,846 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,847 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,847 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 13:56:31,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 13:56:31,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-12 13:56:31,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-12 13:56:31,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-12 13:56:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-12 13:56:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 13:56:31,847 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 13:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. started... 2023-04-12 13:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. as already flushing 2023-04-12 13:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,35663,1681307616979)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-12 13:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-12 13:56:31,853 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,853 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 13:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35663,1681307616979' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-12 13:56:31,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f057907[Count = 0] remaining members to acquire global barrier 2023-04-12 13:56:31,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-12 13:56:31,855 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,856 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-12 13:56:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-12 13:56:31,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,856 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 13:56:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35663,1681307616979' in zk 2023-04-12 13:56:31,859 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,859 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-12 13:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,859 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,859 DEBUG [member: 'jenkins-hbase4.apache.org,35663,1681307616979' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-12 13:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35663,1681307616979': 2023-04-12 13:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35663,1681307616979' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-12 13:56:31,861 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-12 13:56:31,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 13:56:31,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 13:56:31,861 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-12 13:56:31,861 INFO [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 13:56:31,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,863 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 13:56:31,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 13:56:31,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,863 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 13:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,864 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 13:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-12 13:56:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 13:56:31,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,866 DEBUG [(jenkins-hbase4.apache.org,39787,1681307615774)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 13:56:31,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-12 13:56:31,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 13:56:31,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 13:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 13:56:31,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 13:56:31,866 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-12 13:56:31,867 DEBUG [Thread-3704] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-12 13:56:31,867 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:31,867 DEBUG [Thread-3704] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 13:56:31,867 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,867 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-12 13:56:31,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 13:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 13:56:31,867 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-12 13:56:31,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/f1e292c9ae3a4868a89828d2731ba5cf 2023-04-12 13:56:31,967 DEBUG [Thread-3704] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 13:56:31,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 13:56:31,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/7d67b3b25fd348548324bf1c7431c195 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7d67b3b25fd348548324bf1c7431c195 2023-04-12 13:56:31,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7d67b3b25fd348548324bf1c7431c195, entries=200, sequenceid=974, filesize=32.3 K 2023-04-12 13:56:31,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/f81eeb248cbe40b882b0bc3290e2150a as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f81eeb248cbe40b882b0bc3290e2150a 2023-04-12 13:56:31,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f81eeb248cbe40b882b0bc3290e2150a, entries=150, sequenceid=974, filesize=11.9 K 2023-04-12 13:56:31,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/f1e292c9ae3a4868a89828d2731ba5cf as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f1e292c9ae3a4868a89828d2731ba5cf 2023-04-12 13:56:31,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f1e292c9ae3a4868a89828d2731ba5cf, entries=150, sequenceid=974, filesize=11.9 K 2023-04-12 13:56:31,980 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for e35088e2f9a6947abbfc447493169cd6 in 883ms, sequenceid=974, compaction requested=true 2023-04-12 13:56:31,980 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:31,980 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:31,980 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:56:31,980 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:56:31,980 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:31,980 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:31,981 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 119320 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:56:31,981 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49802 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:56:31,981 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/A is initiating minor compaction (all files) 2023-04-12 13:56:31,981 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/B is initiating minor compaction (all files) 2023-04-12 13:56:31,981 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/A in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,981 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/B in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:31,981 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5b9e5f43c84e43da81162ad0371d5b4d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e3930e70ad1e40fba88341e2765a9719, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/dd9f696a6dd6417283de14537d493dd5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7d67b3b25fd348548324bf1c7431c195] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=116.5 K 2023-04-12 13:56:31,981 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/fb284f3f9b004b759c997067e436896c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/06d1d56d65b549de93969eb3e88c375e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6cecd8d3e0fa40d9a8b2da6937f1a040, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f81eeb248cbe40b882b0bc3290e2150a] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=48.6 K 2023-04-12 13:56:31,981 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 5b9e5f43c84e43da81162ad0371d5b4d, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=886, earliestPutTs=1681307788240 2023-04-12 13:56:31,981 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting fb284f3f9b004b759c997067e436896c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681307780850 2023-04-12 13:56:31,982 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting e3930e70ad1e40fba88341e2765a9719, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1681307788985 2023-04-12 13:56:31,982 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 06d1d56d65b549de93969eb3e88c375e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1681307788985 2023-04-12 13:56:31,982 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting dd9f696a6dd6417283de14537d493dd5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681307789252 2023-04-12 13:56:31,982 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting 6cecd8d3e0fa40d9a8b2da6937f1a040, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681307789252 2023-04-12 13:56:31,982 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] compactions.Compactor(207): Compacting 7d67b3b25fd348548324bf1c7431c195, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1681307789955 2023-04-12 13:56:31,982 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f81eeb248cbe40b882b0bc3290e2150a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1681307789955 2023-04-12 13:56:31,990 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#B#compaction#615 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:31,993 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#A#compaction#614 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:32,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35663] regionserver.HRegion(9158): Flush requested on e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:32,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-12 13:56:32,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:32,235 DEBUG [Thread-3696] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:58527 2023-04-12 13:56:32,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:32,235 DEBUG [Thread-3696] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:32,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:32,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:32,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:32,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:32,398 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/ea0b9d9c2be847539b43b42e7ffdb570 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ea0b9d9c2be847539b43b42e7ffdb570 2023-04-12 13:56:32,402 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/B of e35088e2f9a6947abbfc447493169cd6 into ea0b9d9c2be847539b43b42e7ffdb570(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:32,402 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:32,402 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/B, priority=12, startTime=1681307791980; duration=0sec 2023-04-12 13:56:32,402 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 13:56:32,402 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 13:56:32,404 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49802 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-12 13:56:32,404 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1912): e35088e2f9a6947abbfc447493169cd6/C is initiating minor compaction (all files) 2023-04-12 13:56:32,404 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e35088e2f9a6947abbfc447493169cd6/C in TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:32,404 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c8145ed65eed4bdcb6aa3083043083cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/a4b17fcb67ad4413ac7f90612c957d88, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c87b8430af7c41d199ddac1bdc9ab94d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f1e292c9ae3a4868a89828d2731ba5cf] into tmpdir=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp, totalSize=48.6 K 2023-04-12 13:56:32,404 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c8145ed65eed4bdcb6aa3083043083cf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681307780850 2023-04-12 13:56:32,405 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting a4b17fcb67ad4413ac7f90612c957d88, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1681307788985 2023-04-12 13:56:32,405 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting c87b8430af7c41d199ddac1bdc9ab94d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1681307789252 2023-04-12 13:56:32,405 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] compactions.Compactor(207): Compacting f1e292c9ae3a4868a89828d2731ba5cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1681307789955 2023-04-12 13:56:32,412 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] throttle.PressureAwareThroughputController(145): e35088e2f9a6947abbfc447493169cd6#C#compaction#617 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 13:56:32,646 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304120b7b357392624f37bf4d4a2793c51da3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120b7b357392624f37bf4d4a2793c51da3 2023-04-12 13:56:32,807 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/695c103f2c35451199d63309cb449628 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/695c103f2c35451199d63309cb449628 2023-04-12 13:56:32,811 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/A of e35088e2f9a6947abbfc447493169cd6 into 695c103f2c35451199d63309cb449628(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:32,811 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:32,811 INFO [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/A, priority=12, startTime=1681307791980; duration=0sec 2023-04-12 13:56:32,811 DEBUG [RS:0;jenkins-hbase4:35663-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:32,818 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/fdf0d5b4b29a4e39931470a4b28dc8f6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fdf0d5b4b29a4e39931470a4b28dc8f6 2023-04-12 13:56:32,821 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e35088e2f9a6947abbfc447493169cd6/C of e35088e2f9a6947abbfc447493169cd6 into fdf0d5b4b29a4e39931470a4b28dc8f6(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 13:56:32,821 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:32,821 INFO [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6., storeName=e35088e2f9a6947abbfc447493169cd6/C, priority=12, startTime=1681307791980; duration=0sec 2023-04-12 13:56:32,821 DEBUG [RS:0;jenkins-hbase4:35663-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 13:56:33,050 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=984, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/6ef39b80b21a46e29efc95669c3c1ea6 2023-04-12 13:56:33,131 DEBUG [Thread-3694] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:58527 2023-04-12 13:56:33,131 DEBUG [Thread-3694] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:33,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/348591c55bc7434ab924ec0f678f2b08 2023-04-12 13:56:33,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/1da0f56aaf77474881fc7d6dae86d951 2023-04-12 13:56:33,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/6ef39b80b21a46e29efc95669c3c1ea6 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6ef39b80b21a46e29efc95669c3c1ea6 2023-04-12 13:56:33,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6ef39b80b21a46e29efc95669c3c1ea6, entries=100, sequenceid=984, filesize=18.6 K 2023-04-12 13:56:33,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/348591c55bc7434ab924ec0f678f2b08 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/348591c55bc7434ab924ec0f678f2b08 2023-04-12 13:56:33,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/348591c55bc7434ab924ec0f678f2b08, entries=100, sequenceid=984, filesize=9.5 K 2023-04-12 13:56:33,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/1da0f56aaf77474881fc7d6dae86d951 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1da0f56aaf77474881fc7d6dae86d951 2023-04-12 13:56:33,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1da0f56aaf77474881fc7d6dae86d951, entries=100, sequenceid=984, filesize=9.5 K 2023-04-12 13:56:33,880 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for e35088e2f9a6947abbfc447493169cd6 in 1646ms, sequenceid=984, compaction requested=false 2023-04-12 13:56:33,880 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:39,125 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 13:56:40,204 DEBUG [Thread-3698] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:58527 2023-04-12 13:56:40,204 DEBUG [Thread-3698] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:40,211 DEBUG [Thread-3700] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:58527 2023-04-12 13:56:40,211 DEBUG [Thread-3700] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:40,282 DEBUG [Thread-3702] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06799455 to 127.0.0.1:58527 2023-04-12 13:56:40,282 DEBUG [Thread-3702] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 318 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 153 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4205 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4160 2023-04-12 13:56:40,282 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4087 2023-04-12 13:56:40,283 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4212 2023-04-12 13:56:40,283 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(398): read 4169 2023-04-12 13:56:40,283 INFO [Listener at localhost/46495] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-12 13:56:40,283 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:56:40,283 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:58527 2023-04-12 13:56:40,283 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:40,283 INFO [Listener at localhost/46495] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-12 13:56:40,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-12 13:56:40,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:40,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 13:56:40,287 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307800286"}]},"ts":"1681307800286"} 2023-04-12 13:56:40,287 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-12 13:56:40,289 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-12 13:56:40,289 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, UNASSIGN}] 2023-04-12 13:56:40,290 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, UNASSIGN 2023-04-12 13:56:40,291 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:40,291 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307800291"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681307800291"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681307800291"}]},"ts":"1681307800291"} 2023-04-12 13:56:40,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979}] 2023-04-12 13:56:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 13:56:40,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:40,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e35088e2f9a6947abbfc447493169cd6, disabling compactions & flushes 2023-04-12 13:56:40,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:40,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:40,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. after waiting 0 ms 2023-04-12 13:56:40,443 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:40,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e35088e2f9a6947abbfc447493169cd6 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-12 13:56:40,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=A 2023-04-12 13:56:40,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:40,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=B 2023-04-12 13:56:40,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:40,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e35088e2f9a6947abbfc447493169cd6, store=C 2023-04-12 13:56:40,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-12 13:56:40,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 13:56:40,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304127c318901322c4421a809939a7b7d78a8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127c318901322c4421a809939a7b7d78a8 2023-04-12 13:56:40,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 13:56:41,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=994, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/8cfd635790f14acda2dbbc42a6bbf0c2 2023-04-12 13:56:41,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 13:56:41,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/70798a0cf7ec41bb84dc4234921c11dc 2023-04-12 13:56:42,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=994 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/24cbe1d968c84c61aae7000222af57d2 2023-04-12 13:56:42,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/A/8cfd635790f14acda2dbbc42a6bbf0c2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/8cfd635790f14acda2dbbc42a6bbf0c2 2023-04-12 13:56:42,080 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/8cfd635790f14acda2dbbc42a6bbf0c2, entries=100, sequenceid=994, filesize=18.6 K 2023-04-12 13:56:42,081 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/B/70798a0cf7ec41bb84dc4234921c11dc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/70798a0cf7ec41bb84dc4234921c11dc 2023-04-12 13:56:42,084 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/70798a0cf7ec41bb84dc4234921c11dc, entries=100, sequenceid=994, filesize=9.5 K 2023-04-12 13:56:42,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/.tmp/C/24cbe1d968c84c61aae7000222af57d2 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/24cbe1d968c84c61aae7000222af57d2 2023-04-12 13:56:42,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/24cbe1d968c84c61aae7000222af57d2, entries=100, sequenceid=994, filesize=9.5 K 2023-04-12 13:56:42,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e35088e2f9a6947abbfc447493169cd6 in 1644ms, sequenceid=994, compaction requested=true 2023-04-12 13:56:42,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c663ddbb2ad44c58920bfc0b70aa20c9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/18442e2b377a4b659060536064fe486f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c3d03aee6bfb4aee877e0c42742a52fa, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/267b888950994d9d85d865db979f7189, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c138289111db43b0aeec78524a3f7d75, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/50f1a52c50684617b1f27082fe8b1919, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/87859ae4a55c47dc8f0a31c741fc1f32, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3880ddc958a143c099e6f3e935b26d77, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/993babcef0014deab01a0ece083a1615, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d4e02164041e4dfa8bb84f38ca721219, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/bf395584a4cd472a9aaf347b4b1418ed, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5dd2dc5556e547e5a75f6cb551a87369, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f91c89864b684304a221878d4c7abca6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3ad23180649e4f09a712578086113a65, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6c9521f2cf39495b90e3613dc848ac7d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f0ad97f06f5c4c75b20fd0112a9ebaeb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c1f25d2e449e436eb2f21edae1ea8a5f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/804d438e79ff492093af103a8650e97b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d00dc4c2dfc348f3b7454c3255d8ddf1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3245a59bc2654fa8b584a7b64dbb6e1b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/34834dd867924860a7add4ce9763cb1a, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2ee5a90c86674370a27d1005d5de01af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7437133735aa48988312244fe1607d5c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/aa9dbb7ea3914c939cf092c9dbb64c7c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2846ed80b19c4bf782422bb34839f738, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/203887184dad48589d560e0117e303d3, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4f50f971b15549b1b0b8ad4847b0a9a0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/641d801404954837bd1b3e81775697a6, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e0f8c09d8a0645e6bff60bb1e463938c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4424d5d6940e41eab6bf6e5d800e398b, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5b9e5f43c84e43da81162ad0371d5b4d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/ae847923fe004ba1997313af984c2ee4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e3930e70ad1e40fba88341e2765a9719, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/dd9f696a6dd6417283de14537d493dd5, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7d67b3b25fd348548324bf1c7431c195] to archive 2023-04-12 13:56:42,110 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:56:42,112 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c663ddbb2ad44c58920bfc0b70aa20c9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c663ddbb2ad44c58920bfc0b70aa20c9 2023-04-12 13:56:42,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/18442e2b377a4b659060536064fe486f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/18442e2b377a4b659060536064fe486f 2023-04-12 13:56:42,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c3d03aee6bfb4aee877e0c42742a52fa to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c3d03aee6bfb4aee877e0c42742a52fa 2023-04-12 13:56:42,114 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/267b888950994d9d85d865db979f7189 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/267b888950994d9d85d865db979f7189 2023-04-12 13:56:42,115 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c138289111db43b0aeec78524a3f7d75 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c138289111db43b0aeec78524a3f7d75 2023-04-12 13:56:42,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/50f1a52c50684617b1f27082fe8b1919 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/50f1a52c50684617b1f27082fe8b1919 2023-04-12 13:56:42,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/87859ae4a55c47dc8f0a31c741fc1f32 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/87859ae4a55c47dc8f0a31c741fc1f32 2023-04-12 13:56:42,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3880ddc958a143c099e6f3e935b26d77 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3880ddc958a143c099e6f3e935b26d77 2023-04-12 13:56:42,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/993babcef0014deab01a0ece083a1615 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/993babcef0014deab01a0ece083a1615 2023-04-12 13:56:42,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d4e02164041e4dfa8bb84f38ca721219 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d4e02164041e4dfa8bb84f38ca721219 2023-04-12 13:56:42,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/bf395584a4cd472a9aaf347b4b1418ed to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/bf395584a4cd472a9aaf347b4b1418ed 2023-04-12 13:56:42,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5dd2dc5556e547e5a75f6cb551a87369 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5dd2dc5556e547e5a75f6cb551a87369 2023-04-12 13:56:42,122 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f91c89864b684304a221878d4c7abca6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f91c89864b684304a221878d4c7abca6 2023-04-12 13:56:42,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3ad23180649e4f09a712578086113a65 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3ad23180649e4f09a712578086113a65 2023-04-12 13:56:42,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6c9521f2cf39495b90e3613dc848ac7d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6c9521f2cf39495b90e3613dc848ac7d 2023-04-12 13:56:42,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f0ad97f06f5c4c75b20fd0112a9ebaeb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/f0ad97f06f5c4c75b20fd0112a9ebaeb 2023-04-12 13:56:42,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c1f25d2e449e436eb2f21edae1ea8a5f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/c1f25d2e449e436eb2f21edae1ea8a5f 2023-04-12 13:56:42,127 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/804d438e79ff492093af103a8650e97b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/804d438e79ff492093af103a8650e97b 2023-04-12 13:56:42,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d00dc4c2dfc348f3b7454c3255d8ddf1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/d00dc4c2dfc348f3b7454c3255d8ddf1 2023-04-12 13:56:42,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3245a59bc2654fa8b584a7b64dbb6e1b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/3245a59bc2654fa8b584a7b64dbb6e1b 2023-04-12 13:56:42,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/34834dd867924860a7add4ce9763cb1a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/34834dd867924860a7add4ce9763cb1a 2023-04-12 13:56:42,130 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2ee5a90c86674370a27d1005d5de01af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2ee5a90c86674370a27d1005d5de01af 2023-04-12 13:56:42,131 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7437133735aa48988312244fe1607d5c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7437133735aa48988312244fe1607d5c 2023-04-12 13:56:42,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/aa9dbb7ea3914c939cf092c9dbb64c7c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/aa9dbb7ea3914c939cf092c9dbb64c7c 2023-04-12 13:56:42,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2846ed80b19c4bf782422bb34839f738 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/2846ed80b19c4bf782422bb34839f738 2023-04-12 13:56:42,134 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/203887184dad48589d560e0117e303d3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/203887184dad48589d560e0117e303d3 2023-04-12 13:56:42,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4f50f971b15549b1b0b8ad4847b0a9a0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4f50f971b15549b1b0b8ad4847b0a9a0 2023-04-12 13:56:42,136 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/641d801404954837bd1b3e81775697a6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/641d801404954837bd1b3e81775697a6 2023-04-12 13:56:42,137 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e0f8c09d8a0645e6bff60bb1e463938c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e0f8c09d8a0645e6bff60bb1e463938c 2023-04-12 13:56:42,138 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4424d5d6940e41eab6bf6e5d800e398b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/4424d5d6940e41eab6bf6e5d800e398b 2023-04-12 13:56:42,139 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5b9e5f43c84e43da81162ad0371d5b4d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/5b9e5f43c84e43da81162ad0371d5b4d 2023-04-12 13:56:42,139 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/ae847923fe004ba1997313af984c2ee4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/ae847923fe004ba1997313af984c2ee4 2023-04-12 13:56:42,140 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e3930e70ad1e40fba88341e2765a9719 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/e3930e70ad1e40fba88341e2765a9719 2023-04-12 13:56:42,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/dd9f696a6dd6417283de14537d493dd5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/dd9f696a6dd6417283de14537d493dd5 2023-04-12 13:56:42,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7d67b3b25fd348548324bf1c7431c195 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/7d67b3b25fd348548324bf1c7431c195 2023-04-12 13:56:42,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/16cbfdd1cbcb48a08121637f26826480, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8b556f24dc3747d2b836ca8c41306729, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/40dd8774a12048cc9cf61ab7735a29e0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/826c8d5fc2a44c79becabc006b3951cd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/51f1ac7fc1ba4b77a7c23aba92f43802, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/20a85e956d54481bb885ef20f6900091, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6546580a9ef54255b29c73867624e4e0, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8054379535da4b3384b3e5bfa2de090d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/54a9288417ec4669936896b0a37f5d8d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/a2a8dffba135422bbbfc07dca64d66cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/762405ea73214168b4e453cc1d1f7dd1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/4c2baa1173934e058670314032cf65c8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac756d4c75f9430c908362746683a870, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac654c086b094fac8c9f231a1301961f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ed24daacbb4b4a87aef35dab77629521, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f3f9b7f2385c4ecf8670f156863ba4f8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/147fdbd617be4c10b7f1f9818861be8e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f353df5cbab245b693807e778d2c91b9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/bf80b172fe4f47d1b5a81ce12438cdc2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/191de82725c945a394f289a2941b5148, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/fb284f3f9b004b759c997067e436896c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e9ede87c2aa0479f9415cd8910264262, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/87244e7b605d4eaa8e49963a5dd4d8ed, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/1b5b8d5a26e1484fb509402c8c6cbef1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef7c4b439a744e398acfe9d031f7b82e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/3273c2cb89ae47d5970999d018d95952, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef6b924385f84b409ba0181b36574a48, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e2e4096e9de4446bb70167e33c81a178, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/624a1f349e074f8aa204a6742b96e8d8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/22d4f2e8e4c847ae8de437889d23a410, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/0df4f54badbd44519b914d7bfde7e7a4, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/06d1d56d65b549de93969eb3e88c375e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/7b2ef6aa5292475c964bff2f338ebc50, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6cecd8d3e0fa40d9a8b2da6937f1a040, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f81eeb248cbe40b882b0bc3290e2150a] to archive 2023-04-12 13:56:42,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:56:42,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/16cbfdd1cbcb48a08121637f26826480 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/16cbfdd1cbcb48a08121637f26826480 2023-04-12 13:56:42,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8b556f24dc3747d2b836ca8c41306729 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8b556f24dc3747d2b836ca8c41306729 2023-04-12 13:56:42,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/40dd8774a12048cc9cf61ab7735a29e0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/40dd8774a12048cc9cf61ab7735a29e0 2023-04-12 13:56:42,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/826c8d5fc2a44c79becabc006b3951cd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/826c8d5fc2a44c79becabc006b3951cd 2023-04-12 13:56:42,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/51f1ac7fc1ba4b77a7c23aba92f43802 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/51f1ac7fc1ba4b77a7c23aba92f43802 2023-04-12 13:56:42,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/20a85e956d54481bb885ef20f6900091 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/20a85e956d54481bb885ef20f6900091 2023-04-12 13:56:42,152 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6546580a9ef54255b29c73867624e4e0 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6546580a9ef54255b29c73867624e4e0 2023-04-12 13:56:42,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8054379535da4b3384b3e5bfa2de090d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/8054379535da4b3384b3e5bfa2de090d 2023-04-12 13:56:42,153 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/54a9288417ec4669936896b0a37f5d8d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/54a9288417ec4669936896b0a37f5d8d 2023-04-12 13:56:42,154 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/a2a8dffba135422bbbfc07dca64d66cf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/a2a8dffba135422bbbfc07dca64d66cf 2023-04-12 13:56:42,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/762405ea73214168b4e453cc1d1f7dd1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/762405ea73214168b4e453cc1d1f7dd1 2023-04-12 13:56:42,155 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/4c2baa1173934e058670314032cf65c8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/4c2baa1173934e058670314032cf65c8 2023-04-12 13:56:42,156 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac756d4c75f9430c908362746683a870 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac756d4c75f9430c908362746683a870 2023-04-12 13:56:42,157 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac654c086b094fac8c9f231a1301961f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ac654c086b094fac8c9f231a1301961f 2023-04-12 13:56:42,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ed24daacbb4b4a87aef35dab77629521 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ed24daacbb4b4a87aef35dab77629521 2023-04-12 13:56:42,158 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f3f9b7f2385c4ecf8670f156863ba4f8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f3f9b7f2385c4ecf8670f156863ba4f8 2023-04-12 13:56:42,159 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/147fdbd617be4c10b7f1f9818861be8e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/147fdbd617be4c10b7f1f9818861be8e 2023-04-12 13:56:42,160 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f353df5cbab245b693807e778d2c91b9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f353df5cbab245b693807e778d2c91b9 2023-04-12 13:56:42,161 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/bf80b172fe4f47d1b5a81ce12438cdc2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/bf80b172fe4f47d1b5a81ce12438cdc2 2023-04-12 13:56:42,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/191de82725c945a394f289a2941b5148 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/191de82725c945a394f289a2941b5148 2023-04-12 13:56:42,162 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/fb284f3f9b004b759c997067e436896c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/fb284f3f9b004b759c997067e436896c 2023-04-12 13:56:42,163 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e9ede87c2aa0479f9415cd8910264262 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e9ede87c2aa0479f9415cd8910264262 2023-04-12 13:56:42,164 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/87244e7b605d4eaa8e49963a5dd4d8ed to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/87244e7b605d4eaa8e49963a5dd4d8ed 2023-04-12 13:56:42,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/1b5b8d5a26e1484fb509402c8c6cbef1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/1b5b8d5a26e1484fb509402c8c6cbef1 2023-04-12 13:56:42,165 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef7c4b439a744e398acfe9d031f7b82e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef7c4b439a744e398acfe9d031f7b82e 2023-04-12 13:56:42,166 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/3273c2cb89ae47d5970999d018d95952 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/3273c2cb89ae47d5970999d018d95952 2023-04-12 13:56:42,167 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef6b924385f84b409ba0181b36574a48 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ef6b924385f84b409ba0181b36574a48 2023-04-12 13:56:42,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e2e4096e9de4446bb70167e33c81a178 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/e2e4096e9de4446bb70167e33c81a178 2023-04-12 13:56:42,168 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/624a1f349e074f8aa204a6742b96e8d8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/624a1f349e074f8aa204a6742b96e8d8 2023-04-12 13:56:42,169 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/22d4f2e8e4c847ae8de437889d23a410 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/22d4f2e8e4c847ae8de437889d23a410 2023-04-12 13:56:42,170 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/0df4f54badbd44519b914d7bfde7e7a4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/0df4f54badbd44519b914d7bfde7e7a4 2023-04-12 13:56:42,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/06d1d56d65b549de93969eb3e88c375e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/06d1d56d65b549de93969eb3e88c375e 2023-04-12 13:56:42,171 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/7b2ef6aa5292475c964bff2f338ebc50 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/7b2ef6aa5292475c964bff2f338ebc50 2023-04-12 13:56:42,172 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6cecd8d3e0fa40d9a8b2da6937f1a040 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/6cecd8d3e0fa40d9a8b2da6937f1a040 2023-04-12 13:56:42,173 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f81eeb248cbe40b882b0bc3290e2150a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/f81eeb248cbe40b882b0bc3290e2150a 2023-04-12 13:56:42,175 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8df74b873c97491cab54f68c7ffc2a76, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/615721d738bf483c9b6a88dc651e8612, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7618be663d7d4cb7be93db695ca96654, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8249707d921d464a92d8ac2331152e84, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f6bab67dc6cf4353ad896b961f063574, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8445e94adccf4d9aae8a40efeac2bf68, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/02efa948545d4d5ab916f6f7c7f325a1, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/ca2815cbfb0742ceaa3c1090c8208951, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/04a70e82f7394d3db0c506eff032a9af, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/752e2b88a5a348e9a608ffc19ee4a040, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d3467cb1795842968e302127ee92f5b9, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/03f2ed7bacc9435f81a5b6b40e1f0472, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d50cfced1489458bb5bc3ad5a6a7f2fb, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/58001a5e2cc8454783ddb9821a7377f7, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f87dd1b87e5844f78c5871b8148d1574, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e839b7826218411fb9c3240af0ee4d58, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/721264f6e735469b9a4f726e85e7ab4d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/b63899f348aa4745b6fa0fa921960a2e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c262ced90e404f399d37c29240fb86f2, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/42a47ff7c2e245cfa6979a9635310d2c, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c8145ed65eed4bdcb6aa3083043083cf, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9947f535ea124ae08d7b69f4ef794305, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/76c4b8663547447d97a992f917488837, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/529c2133edbb45e89e8da34df605ae0f, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9260b20268cf4c8091c8b18efc046d46, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/3ae64a42aab847a899ae3792ae60cbfd, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7490a41ff9b14eecaa55ded26e41d260, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1095d91620874c66be5bb9f14ae3fb67, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/372bf1b75ce045c3824f502896c48309, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e9aeeabd41aa470ca50562a1d0fa44e8, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fbd769cb8adf450c832b9a9bcc0c785e, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/a4b17fcb67ad4413ac7f90612c957d88, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/862818e76e104dfab40d3b3344713302, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c87b8430af7c41d199ddac1bdc9ab94d, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f1e292c9ae3a4868a89828d2731ba5cf] to archive 2023-04-12 13:56:42,176 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 13:56:42,177 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8df74b873c97491cab54f68c7ffc2a76 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8df74b873c97491cab54f68c7ffc2a76 2023-04-12 13:56:42,178 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/615721d738bf483c9b6a88dc651e8612 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/615721d738bf483c9b6a88dc651e8612 2023-04-12 13:56:42,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7618be663d7d4cb7be93db695ca96654 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7618be663d7d4cb7be93db695ca96654 2023-04-12 13:56:42,179 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8249707d921d464a92d8ac2331152e84 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8249707d921d464a92d8ac2331152e84 2023-04-12 13:56:42,180 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f6bab67dc6cf4353ad896b961f063574 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f6bab67dc6cf4353ad896b961f063574 2023-04-12 13:56:42,181 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8445e94adccf4d9aae8a40efeac2bf68 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/8445e94adccf4d9aae8a40efeac2bf68 2023-04-12 13:56:42,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/02efa948545d4d5ab916f6f7c7f325a1 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/02efa948545d4d5ab916f6f7c7f325a1 2023-04-12 13:56:42,182 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/ca2815cbfb0742ceaa3c1090c8208951 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/ca2815cbfb0742ceaa3c1090c8208951 2023-04-12 13:56:42,183 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/04a70e82f7394d3db0c506eff032a9af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/04a70e82f7394d3db0c506eff032a9af 2023-04-12 13:56:42,184 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/752e2b88a5a348e9a608ffc19ee4a040 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/752e2b88a5a348e9a608ffc19ee4a040 2023-04-12 13:56:42,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d3467cb1795842968e302127ee92f5b9 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d3467cb1795842968e302127ee92f5b9 2023-04-12 13:56:42,185 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/03f2ed7bacc9435f81a5b6b40e1f0472 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/03f2ed7bacc9435f81a5b6b40e1f0472 2023-04-12 13:56:42,186 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d50cfced1489458bb5bc3ad5a6a7f2fb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/d50cfced1489458bb5bc3ad5a6a7f2fb 2023-04-12 13:56:42,187 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/58001a5e2cc8454783ddb9821a7377f7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/58001a5e2cc8454783ddb9821a7377f7 2023-04-12 13:56:42,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f87dd1b87e5844f78c5871b8148d1574 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f87dd1b87e5844f78c5871b8148d1574 2023-04-12 13:56:42,188 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e839b7826218411fb9c3240af0ee4d58 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e839b7826218411fb9c3240af0ee4d58 2023-04-12 13:56:42,189 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/721264f6e735469b9a4f726e85e7ab4d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/721264f6e735469b9a4f726e85e7ab4d 2023-04-12 13:56:42,190 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/b63899f348aa4745b6fa0fa921960a2e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/b63899f348aa4745b6fa0fa921960a2e 2023-04-12 13:56:42,191 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c262ced90e404f399d37c29240fb86f2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c262ced90e404f399d37c29240fb86f2 2023-04-12 13:56:42,192 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/42a47ff7c2e245cfa6979a9635310d2c to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/42a47ff7c2e245cfa6979a9635310d2c 2023-04-12 13:56:42,193 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c8145ed65eed4bdcb6aa3083043083cf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c8145ed65eed4bdcb6aa3083043083cf 2023-04-12 13:56:42,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9947f535ea124ae08d7b69f4ef794305 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9947f535ea124ae08d7b69f4ef794305 2023-04-12 13:56:42,194 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/76c4b8663547447d97a992f917488837 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/76c4b8663547447d97a992f917488837 2023-04-12 13:56:42,195 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/529c2133edbb45e89e8da34df605ae0f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/529c2133edbb45e89e8da34df605ae0f 2023-04-12 13:56:42,196 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9260b20268cf4c8091c8b18efc046d46 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/9260b20268cf4c8091c8b18efc046d46 2023-04-12 13:56:42,197 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/3ae64a42aab847a899ae3792ae60cbfd to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/3ae64a42aab847a899ae3792ae60cbfd 2023-04-12 13:56:42,198 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7490a41ff9b14eecaa55ded26e41d260 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/7490a41ff9b14eecaa55ded26e41d260 2023-04-12 13:56:42,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1095d91620874c66be5bb9f14ae3fb67 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1095d91620874c66be5bb9f14ae3fb67 2023-04-12 13:56:42,199 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/372bf1b75ce045c3824f502896c48309 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/372bf1b75ce045c3824f502896c48309 2023-04-12 13:56:42,200 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e9aeeabd41aa470ca50562a1d0fa44e8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/e9aeeabd41aa470ca50562a1d0fa44e8 2023-04-12 13:56:42,201 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fbd769cb8adf450c832b9a9bcc0c785e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fbd769cb8adf450c832b9a9bcc0c785e 2023-04-12 13:56:42,202 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/a4b17fcb67ad4413ac7f90612c957d88 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/a4b17fcb67ad4413ac7f90612c957d88 2023-04-12 13:56:42,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/862818e76e104dfab40d3b3344713302 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/862818e76e104dfab40d3b3344713302 2023-04-12 13:56:42,203 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c87b8430af7c41d199ddac1bdc9ab94d to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/c87b8430af7c41d199ddac1bdc9ab94d 2023-04-12 13:56:42,204 DEBUG [StoreCloser-TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f1e292c9ae3a4868a89828d2731ba5cf to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/f1e292c9ae3a4868a89828d2731ba5cf 2023-04-12 13:56:42,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/recovered.edits/997.seqid, newMaxSeqId=997, maxSeqId=4 2023-04-12 13:56:42,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6. 2023-04-12 13:56:42,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e35088e2f9a6947abbfc447493169cd6: 2023-04-12 13:56:42,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:42,210 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=e35088e2f9a6947abbfc447493169cd6, regionState=CLOSED 2023-04-12 13:56:42,210 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681307802210"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681307802210"}]},"ts":"1681307802210"} 2023-04-12 13:56:42,212 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-12 13:56:42,212 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure e35088e2f9a6947abbfc447493169cd6, server=jenkins-hbase4.apache.org,35663,1681307616979 in 1.9190 sec 2023-04-12 13:56:42,213 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-12 13:56:42,213 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e35088e2f9a6947abbfc447493169cd6, UNASSIGN in 1.9230 sec 2023-04-12 13:56:42,214 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681307802214"}]},"ts":"1681307802214"} 2023-04-12 13:56:42,214 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-12 13:56:42,230 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-12 13:56:42,231 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9460 sec 2023-04-12 13:56:42,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-12 13:56:42,391 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-12 13:56:42,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-12 13:56:42,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:42,393 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:42,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-12 13:56:42,394 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:42,398 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:42,399 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C, FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/recovered.edits] 2023-04-12 13:56:42,402 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/695c103f2c35451199d63309cb449628 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/695c103f2c35451199d63309cb449628 2023-04-12 13:56:42,402 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6ef39b80b21a46e29efc95669c3c1ea6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/6ef39b80b21a46e29efc95669c3c1ea6 2023-04-12 13:56:42,403 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/8cfd635790f14acda2dbbc42a6bbf0c2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/A/8cfd635790f14acda2dbbc42a6bbf0c2 2023-04-12 13:56:42,405 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/348591c55bc7434ab924ec0f678f2b08 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/348591c55bc7434ab924ec0f678f2b08 2023-04-12 13:56:42,406 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/70798a0cf7ec41bb84dc4234921c11dc to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/70798a0cf7ec41bb84dc4234921c11dc 2023-04-12 13:56:42,407 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ea0b9d9c2be847539b43b42e7ffdb570 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/B/ea0b9d9c2be847539b43b42e7ffdb570 2023-04-12 13:56:42,409 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1da0f56aaf77474881fc7d6dae86d951 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/1da0f56aaf77474881fc7d6dae86d951 2023-04-12 13:56:42,410 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/24cbe1d968c84c61aae7000222af57d2 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/24cbe1d968c84c61aae7000222af57d2 2023-04-12 13:56:42,411 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fdf0d5b4b29a4e39931470a4b28dc8f6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/C/fdf0d5b4b29a4e39931470a4b28dc8f6 2023-04-12 13:56:42,413 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/recovered.edits/997.seqid to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6/recovered.edits/997.seqid 2023-04-12 13:56:42,414 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/.tmp/data/default/TestAcidGuarantees/e35088e2f9a6947abbfc447493169cd6 2023-04-12 13:56:42,414 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-12 13:56:42,414 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 13:56:42,415 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-12 13:56:42,417 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120b7b357392624f37bf4d4a2793c51da3 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304120b7b357392624f37bf4d4a2793c51da3 2023-04-12 13:56:42,418 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412157293e289154e4f99ba4df615cd465a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412157293e289154e4f99ba4df615cd465a 2023-04-12 13:56:42,419 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041217ebc7dcaf16400d8076b0576aab22eb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041217ebc7dcaf16400d8076b0576aab22eb 2023-04-12 13:56:42,420 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121d7993aef228495b9d86ec183616db81 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304121d7993aef228495b9d86ec183616db81 2023-04-12 13:56:42,421 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122d1c0ecacdd345ae941316cd3bd27899 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304122d1c0ecacdd345ae941316cd3bd27899 2023-04-12 13:56:42,422 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041235b8065a4fb6402887652cb431c9e28b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041235b8065a4fb6402887652cb431c9e28b 2023-04-12 13:56:42,423 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041244992b72b20247e1aebb8d63cf2b56f7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041244992b72b20247e1aebb8d63cf2b56f7 2023-04-12 13:56:42,423 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124600869cf7434b82bf8c2a8b143359af to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124600869cf7434b82bf8c2a8b143359af 2023-04-12 13:56:42,424 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124aacdb33488e4ec9967a991f53035fb4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124aacdb33488e4ec9967a991f53035fb4 2023-04-12 13:56:42,425 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124e1a1acb709445f8834b3a9dc360b736 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304124e1a1acb709445f8834b3a9dc360b736 2023-04-12 13:56:42,426 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125476da05cefb4450a85e8abf847e2b8e to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125476da05cefb4450a85e8abf847e2b8e 2023-04-12 13:56:42,428 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041256e7a1af237149c6850464c973b83af4 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041256e7a1af237149c6850464c973b83af4 2023-04-12 13:56:42,429 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125ba9048260b84978a913d2f28be6828b to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125ba9048260b84978a913d2f28be6828b 2023-04-12 13:56:42,430 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125f4aef7b4c7d4a44b40eca159cd0cafb to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304125f4aef7b4c7d4a44b40eca159cd0cafb 2023-04-12 13:56:42,431 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412667917b413524752bb822b807ac00258 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412667917b413524752bb822b807ac00258 2023-04-12 13:56:42,432 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041266cbaaddb42940f99ae8819a7f1d8010 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041266cbaaddb42940f99ae8819a7f1d8010 2023-04-12 13:56:42,433 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127c318901322c4421a809939a7b7d78a8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304127c318901322c4421a809939a7b7d78a8 2023-04-12 13:56:42,433 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041280b4b98b14194aa3ba9a5a5694747321 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041280b4b98b14194aa3ba9a5a5694747321 2023-04-12 13:56:42,434 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412832c1ec197f442a6b504ca9ff82f43b7 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412832c1ec197f442a6b504ca9ff82f43b7 2023-04-12 13:56:42,435 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412864719f00fd34aacbc035ff1b1910070 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412864719f00fd34aacbc035ff1b1910070 2023-04-12 13:56:42,436 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128e29f512004443fa8a23d6a90cd770e8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128e29f512004443fa8a23d6a90cd770e8 2023-04-12 13:56:42,437 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128f2deb01819f4eab9246e2dfecfc3324 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304128f2deb01819f4eab9246e2dfecfc3324 2023-04-12 13:56:42,438 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129fb6a34ebc1f44928448f45ad81ea109 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304129fb6a34ebc1f44928448f45ad81ea109 2023-04-12 13:56:42,438 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a165123ef3d34f56a6ceef9de8cf98a5 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a165123ef3d34f56a6ceef9de8cf98a5 2023-04-12 13:56:42,439 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a301df2b56134abb944b56712ef5e344 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a301df2b56134abb944b56712ef5e344 2023-04-12 13:56:42,440 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a6c4f59384004ae0811a0696d4a5aa1a to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412a6c4f59384004ae0811a0696d4a5aa1a 2023-04-12 13:56:42,441 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b6c51390494748abb6702fee28eba76f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b6c51390494748abb6702fee28eba76f 2023-04-12 13:56:42,442 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b74581366e0643349d1bbedee0490d2f to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412b74581366e0643349d1bbedee0490d2f 2023-04-12 13:56:42,443 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d6c9f70a326148eea782ef3b816ff3c8 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412d6c9f70a326148eea782ef3b816ff3c8 2023-04-12 13:56:42,444 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e5a4c8c2b82844efb88e6221b939c2d6 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412e5a4c8c2b82844efb88e6221b939c2d6 2023-04-12 13:56:42,445 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fbb29e3dd636483dabe9ec25cf8f6825 to hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230412fbb29e3dd636483dabe9ec25cf8f6825 2023-04-12 13:56:42,445 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-12 13:56:42,447 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:42,448 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-12 13:56:42,450 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-12 13:56:42,450 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:42,451 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-12 13:56:42,451 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681307802451"}]},"ts":"9223372036854775807"} 2023-04-12 13:56:42,452 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 13:56:42,452 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e35088e2f9a6947abbfc447493169cd6, NAME => 'TestAcidGuarantees,,1681307769148.e35088e2f9a6947abbfc447493169cd6.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 13:56:42,452 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-12 13:56:42,452 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681307802452"}]},"ts":"9223372036854775807"} 2023-04-12 13:56:42,453 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-12 13:56:42,455 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-12 13:56:42,456 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-12 13:56:42,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39787] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-12 13:56:42,495 INFO [Listener at localhost/46495] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-12 13:56:42,506 INFO [Listener at localhost/46495] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 229), OpenFileDescriptor=531 (was 572), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=426 (was 415) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=868 (was 771) - AvailableMemoryMB LEAK? - 2023-04-12 13:56:42,506 INFO [Listener at localhost/46495] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-12 13:56:42,506 INFO [Listener at localhost/46495] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 13:56:42,507 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:58527 2023-04-12 13:56:42,507 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:42,507 DEBUG [Listener at localhost/46495] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-12 13:56:42,508 DEBUG [Listener at localhost/46495] util.JVMClusterUtil(257): Found active master hash=522626403, stopped=false 2023-04-12 13:56:42,508 INFO [Listener at localhost/46495] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:56:42,513 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 13:56:42,513 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 13:56:42,513 INFO [Listener at localhost/46495] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 13:56:42,513 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:56:42,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 13:56:42,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 13:56:42,514 DEBUG [Listener at localhost/46495] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60f248a7 to 127.0.0.1:58527 2023-04-12 13:56:42,514 DEBUG [Listener at localhost/46495] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:42,514 INFO [Listener at localhost/46495] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35663,1681307616979' ***** 2023-04-12 13:56:42,514 INFO [Listener at localhost/46495] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-12 13:56:42,514 INFO [RS:0;jenkins-hbase4:35663] regionserver.HeapMemoryManager(220): Stopping 2023-04-12 13:56:42,515 INFO [RS:0;jenkins-hbase4:35663] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-12 13:56:42,515 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-12 13:56:42,515 INFO [RS:0;jenkins-hbase4:35663] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-12 13:56:42,515 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(3304): Received CLOSE for 083bda99a45a5ebe3c6fdff5c36223cb 2023-04-12 13:56:42,515 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:42,515 DEBUG [RS:0;jenkins-hbase4:35663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79c8271d to 127.0.0.1:58527 2023-04-12 13:56:42,515 DEBUG [RS:0;jenkins-hbase4:35663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:42,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 083bda99a45a5ebe3c6fdff5c36223cb, disabling compactions & flushes 2023-04-12 13:56:42,516 INFO [RS:0;jenkins-hbase4:35663] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-12 13:56:42,516 INFO [RS:0;jenkins-hbase4:35663] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-12 13:56:42,516 INFO [RS:0;jenkins-hbase4:35663] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-12 13:56:42,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:56:42,516 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-12 13:56:42,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:56:42,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. after waiting 0 ms 2023-04-12 13:56:42,516 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:56:42,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 083bda99a45a5ebe3c6fdff5c36223cb 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-12 13:56:42,516 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-12 13:56:42,517 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-12 13:56:42,517 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1479): Online Regions={083bda99a45a5ebe3c6fdff5c36223cb=hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb., 1588230740=hbase:meta,,1.1588230740} 2023-04-12 13:56:42,517 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-12 13:56:42,517 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-12 13:56:42,517 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-12 13:56:42,517 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-12 13:56:42,517 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-12 13:56:42,519 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1505): Waiting on 083bda99a45a5ebe3c6fdff5c36223cb, 1588230740 2023-04-12 13:56:42,529 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/.tmp/info/06674d0b674c4fa3a941315e8ec3b277 2023-04-12 13:56:42,533 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06674d0b674c4fa3a941315e8ec3b277 2023-04-12 13:56:42,555 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-12 13:56:42,719 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1505): Waiting on 083bda99a45a5ebe3c6fdff5c36223cb, 1588230740 2023-04-12 13:56:42,919 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1505): Waiting on 083bda99a45a5ebe3c6fdff5c36223cb, 1588230740 2023-04-12 13:56:42,926 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/.tmp/info/a97cebfdbbdc457fb65763b7c39f5fd5 2023-04-12 13:56:42,930 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/.tmp/info/a97cebfdbbdc457fb65763b7c39f5fd5 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/info/a97cebfdbbdc457fb65763b7c39f5fd5 2023-04-12 13:56:42,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/info/a97cebfdbbdc457fb65763b7c39f5fd5, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 13:56:42,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 083bda99a45a5ebe3c6fdff5c36223cb in 418ms, sequenceid=6, compaction requested=false 2023-04-12 13:56:42,938 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/.tmp/rep_barrier/8fb3b772dc9a41f4a63e9eb760d89ffc 2023-04-12 13:56:42,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/namespace/083bda99a45a5ebe3c6fdff5c36223cb/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-12 13:56:42,940 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:56:42,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 083bda99a45a5ebe3c6fdff5c36223cb: 2023-04-12 13:56:42,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681307619767.083bda99a45a5ebe3c6fdff5c36223cb. 2023-04-12 13:56:42,941 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8fb3b772dc9a41f4a63e9eb760d89ffc 2023-04-12 13:56:43,120 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-12 13:56:43,208 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-12 13:56:43,208 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-12 13:56:43,320 DEBUG [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-12 13:56:43,346 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/.tmp/table/01d242677fb144e2ac82186e66166de9 2023-04-12 13:56:43,349 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 01d242677fb144e2ac82186e66166de9 2023-04-12 13:56:43,350 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/.tmp/info/06674d0b674c4fa3a941315e8ec3b277 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/info/06674d0b674c4fa3a941315e8ec3b277 2023-04-12 13:56:43,352 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06674d0b674c4fa3a941315e8ec3b277 2023-04-12 13:56:43,353 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/info/06674d0b674c4fa3a941315e8ec3b277, entries=22, sequenceid=93, filesize=7.1 K 2023-04-12 13:56:43,354 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/.tmp/rep_barrier/8fb3b772dc9a41f4a63e9eb760d89ffc as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/rep_barrier/8fb3b772dc9a41f4a63e9eb760d89ffc 2023-04-12 13:56:43,357 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8fb3b772dc9a41f4a63e9eb760d89ffc 2023-04-12 13:56:43,357 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/rep_barrier/8fb3b772dc9a41f4a63e9eb760d89ffc, entries=6, sequenceid=93, filesize=5.5 K 2023-04-12 13:56:43,358 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/.tmp/table/01d242677fb144e2ac82186e66166de9 as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/table/01d242677fb144e2ac82186e66166de9 2023-04-12 13:56:43,360 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 01d242677fb144e2ac82186e66166de9 2023-04-12 13:56:43,360 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/table/01d242677fb144e2ac82186e66166de9, entries=9, sequenceid=93, filesize=5.5 K 2023-04-12 13:56:43,361 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 844ms, sequenceid=93, compaction requested=false 2023-04-12 13:56:43,369 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-12 13:56:43,369 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-12 13:56:43,370 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-12 13:56:43,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-12 13:56:43,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-12 13:56:43,520 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35663,1681307616979; all regions closed. 2023-04-12 13:56:43,528 DEBUG [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/oldWALs 2023-04-12 13:56:43,528 INFO [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35663%2C1681307616979.meta:.meta(num 1681307619490) 2023-04-12 13:56:43,532 DEBUG [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/oldWALs 2023-04-12 13:56:43,532 INFO [RS:0;jenkins-hbase4:35663] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35663%2C1681307616979:(num 1681307619285) 2023-04-12 13:56:43,533 DEBUG [RS:0;jenkins-hbase4:35663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:43,533 INFO [RS:0;jenkins-hbase4:35663] regionserver.LeaseManager(133): Closed leases 2023-04-12 13:56:43,533 INFO [RS:0;jenkins-hbase4:35663] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-12 13:56:43,533 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 13:56:43,535 INFO [RS:0;jenkins-hbase4:35663] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35663 2023-04-12 13:56:43,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35663,1681307616979 2023-04-12 13:56:43,540 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 13:56:43,540 ERROR [Listener at localhost/46495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6c36823d rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2098] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1374) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 13:56:43,541 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 13:56:43,541 ERROR [Listener at localhost/46495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4493844c rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2098] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 13:56:43,542 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35663,1681307616979] 2023-04-12 13:56:43,543 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35663,1681307616979; numProcessing=1 2023-04-12 13:56:43,544 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35663,1681307616979 already deleted, retry=false 2023-04-12 13:56:43,544 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35663,1681307616979 expired; onlineServers=0 2023-04-12 13:56:43,544 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39787,1681307615774' ***** 2023-04-12 13:56:43,544 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-12 13:56:43,544 DEBUG [M:0;jenkins-hbase4:39787] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e1baf58, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 13:56:43,544 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39787,1681307615774 2023-04-12 13:56:43,544 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39787,1681307615774; all regions closed. 2023-04-12 13:56:43,544 DEBUG [M:0;jenkins-hbase4:39787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 13:56:43,544 DEBUG [M:0;jenkins-hbase4:39787] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-12 13:56:43,544 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-12 13:56:43,544 DEBUG [M:0;jenkins-hbase4:39787] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-12 13:56:43,544 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681307618984] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681307618984,5,FailOnTimeoutGroup] 2023-04-12 13:56:43,544 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681307618984] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681307618984,5,FailOnTimeoutGroup] 2023-04-12 13:56:43,544 INFO [M:0;jenkins-hbase4:39787] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-12 13:56:43,545 INFO [M:0;jenkins-hbase4:39787] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-12 13:56:43,545 INFO [M:0;jenkins-hbase4:39787] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-12 13:56:43,545 DEBUG [M:0;jenkins-hbase4:39787] master.HMaster(1512): Stopping service threads 2023-04-12 13:56:43,545 INFO [M:0;jenkins-hbase4:39787] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-12 13:56:43,545 ERROR [M:0;jenkins-hbase4:39787] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-12 13:56:43,546 INFO [M:0;jenkins-hbase4:39787] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 13:56:43,546 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-12 13:56:43,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-12 13:56:43,546 DEBUG [M:0;jenkins-hbase4:39787] zookeeper.ZKUtil(398): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-12 13:56:43,546 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 13:56:43,546 WARN [M:0;jenkins-hbase4:39787] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-12 13:56:43,547 INFO [M:0;jenkins-hbase4:39787] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 13:56:43,547 INFO [M:0;jenkins-hbase4:39787] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-12 13:56:43,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 13:56:43,547 DEBUG [M:0;jenkins-hbase4:39787] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-12 13:56:43,547 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:56:43,547 DEBUG [M:0;jenkins-hbase4:39787] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:56:43,547 DEBUG [M:0;jenkins-hbase4:39787] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-12 13:56:43,547 DEBUG [M:0;jenkins-hbase4:39787] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:56:43,547 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.85 KB heapSize=393.61 KB 2023-04-12 13:56:43,643 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 13:56:43,643 INFO [RS:0;jenkins-hbase4:35663] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35663,1681307616979; zookeeper connection closed. 2023-04-12 13:56:43,643 ERROR [Listener at localhost/46495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3c1e9374 rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2098] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 13:56:43,644 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): regionserver:35663-0x10178a628840001, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 13:56:43,644 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3efd71e5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3efd71e5 2023-04-12 13:56:43,644 ERROR [Listener at localhost/46495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@154d3b45 rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2098] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 13:56:43,645 INFO [Listener at localhost/46495] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-12 13:56:43,958 INFO [M:0;jenkins-hbase4:39787] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.85 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/36fd95a3bffc4580bc1648c0882d6ede 2023-04-12 13:56:43,962 INFO [M:0;jenkins-hbase4:39787] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 36fd95a3bffc4580bc1648c0882d6ede 2023-04-12 13:56:43,963 DEBUG [M:0;jenkins-hbase4:39787] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/36fd95a3bffc4580bc1648c0882d6ede as hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/36fd95a3bffc4580bc1648c0882d6ede 2023-04-12 13:56:43,966 INFO [M:0;jenkins-hbase4:39787] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 36fd95a3bffc4580bc1648c0882d6ede 2023-04-12 13:56:43,966 INFO [M:0;jenkins-hbase4:39787] regionserver.HStore(1080): Added hdfs://localhost:34129/user/jenkins/test-data/90820405-51fd-7d8b-31f5-02d51653b964/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/36fd95a3bffc4580bc1648c0882d6ede, entries=65, sequenceid=578, filesize=24.7 K 2023-04-12 13:56:43,966 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegion(2948): Finished flush of dataSize ~344.85 KB/353126, heapSize ~393.59 KB/403040, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=578, compaction requested=false 2023-04-12 13:56:43,968 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 13:56:43,968 DEBUG [M:0;jenkins-hbase4:39787] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 13:56:43,971 INFO [M:0;jenkins-hbase4:39787] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-12 13:56:43,971 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 13:56:43,972 INFO [M:0;jenkins-hbase4:39787] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39787 2023-04-12 13:56:43,974 DEBUG [M:0;jenkins-hbase4:39787] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,39787,1681307615774 already deleted, retry=false 2023-04-12 13:56:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 13:56:44,076 INFO [M:0;jenkins-hbase4:39787] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39787,1681307615774; zookeeper connection closed. 2023-04-12 13:56:44,076 ERROR [Listener at localhost/46495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@45ea0f5f rejected from java.util.concurrent.ThreadPoolExecutor@463f8130[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 13:56:44,076 DEBUG [Listener at localhost/46495-EventThread] zookeeper.ZKWatcher(600): master:39787-0x10178a628840000, quorum=127.0.0.1:58527, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 13:56:44,076 ERROR [Listener at localhost/46495-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@72b26af3 rejected from java.util.concurrent.ThreadPoolExecutor@463f8130[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 13:56:44,078 WARN [Listener at localhost/46495] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-12 13:56:44,080 INFO [Listener at localhost/46495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 13:56:44,188 WARN [BP-1838550249-172.31.14.131-1681307612384 heartbeating to localhost/127.0.0.1:34129] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-12 13:56:44,188 WARN [BP-1838550249-172.31.14.131-1681307612384 heartbeating to localhost/127.0.0.1:34129] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1838550249-172.31.14.131-1681307612384 (Datanode Uuid 06037e37-fbd1-4b2b-b8f8-c56f07685505) service to localhost/127.0.0.1:34129 2023-04-12 13:56:44,190 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/cluster_34daed27-f3ec-9fef-aef6-6c33478889d6/dfs/data/data1/current/BP-1838550249-172.31.14.131-1681307612384] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 13:56:44,191 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b0187529-45a4-e442-cf0e-34c484e4ec00/cluster_34daed27-f3ec-9fef-aef6-6c33478889d6/dfs/data/data2/current/BP-1838550249-172.31.14.131-1681307612384] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 13:56:44,225 INFO [Listener at localhost/46495] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 13:56:44,392 INFO [Listener at localhost/46495] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-12 13:56:44,561 INFO [Listener at localhost/46495] hbase.HBaseTestingUtility(1293): Minicluster is down